This paper proposes a process that supports the full life-cycle to generate and exploit knowledge graphs (KG) from Earth observation rasters and open data. The innovative features of this process include i) an algorithm to automatically identify Regions of Interest (ROIs) on a raster, o ering an accurate geographic division, i.e. geolocated references used to guide data integration; ii) a semantic-driven process to generate a KG from di erent sources (raster, open, linked and social data); and iii) a validation of the approach with a use case on re detection, which shows the added value of KGs to identify ROIs where high changes have been detected.
Thanks to e cient deep learning models, Earth Observation satellite images (EO
images) can be more easily compared so that changes between series of two or
more images can be computed. Changes are indicators that may reveal natural
events or disasters such as res, oods, or earthquakes; changes in vegetation like
frost impact on cultures, or damaged forests after hurricanes; or long term
evolution of ground occupation like deforestation, urbanisation, etc. Services assessing
geographic changes over time typically require searching, discovering, and
manually identifying relevant data. This is a di cult and laborious task that relies on
domain knowledge and that is often subject to human error [21]. Automatically
identifying changes thanks to EO satellite image processing reduces this e ort.
Whereas most early works relied on supervised ML algorithms [20], more recent
ones are unsupervised to avoid manual tagging of examples before training their
models [
As an example, let's consider Figures 1(a) and 1(b), which depict snapshots of the same area a ected by an important re, that can hardly be detected by human eye when comparing the two images. Automatic change detection algorithms are much more e cient, as seen in Figure 1(b) where change areas appear in white. Generic change algorithms are e cient to detect changes, to assign them a degree of importance (high, medium, low) but they do not identify the nature of the event that caused the change. To do so, more contextual data is needed. Various kinds of geospatial data can provide context to detect changes and highlight speci c kinds of phenomenon: either indices (e.g., vegetation, hydrography or landscape indices) directly computed from EO images; or linked data and social data used as rich complementary sources of information. Providing contextual knowledge to explain or annotate "what can be seen" or "what can be interpreted" on EO images has been the goal of many projects for the past 15 years. This process requires the integration of data from various sources and of di erent nature, which can be e ciently done thanks to semantic web technologies.
Unlike most of previous works that deal with a single type of event (i.e. ood [16] or wild res [17]), we wanted to design a generic process so that (i) given the type of events to be identi ed from the change impact of EO images, any kind of dataset could be selected and ingested as semantic contextual knowledge, (ii) it could take as input any change raster whatever the algorithm that generated it; (iii) it could manage the geometry of events with a shape that could reach a compromise between precision (to know the event exact location) and simplicity (to make it easy to compare with the geometry of other geographic objects). The task we address here can be formulated as such: given the results of an unsupervised change detection algorithm on an area of interest, we want to build a knowledge graph (KG) depicting various contextual data related to each event of a speci c kind identi able on that region at the considered period. One di culty we address here is to optimally group pixels as for de ning relevant regions, i.e., polygons representing Regions Of Interest (ROIs) and to nd the accurate geographic division that will guide data integration. Similar to the notion of tiling grid de ned by ESA, that divides the Earth surface into tiles representing a xed area on this surface, the notion of ROI is convenient to be used as a geolocated reference for data integration. Unlike tiles, not all ROIs have the same size or shape. The innovative features of the proposed process include a) an algorithm to identify ROIs on a raster; b) a semantic-driven process and a vocabulary to generate a KG from di erent sources in order to help explaining the changes identi ed in change rasters; iii) a validation of the approach with use-cases on re detection, which shows the added value of KGs to explain automatically ROIs where high changes have been detected.
In the following, Section 2 overviews the approach and details the KG generation process. Section 3 describes its evaluation in terms of change explanation applied to a use-case. Section 4 discusses the main related works. Finally, Section 5 concludes the paper and presents future work. 2
The process that we propose takes as input a raster le providing geospatial data, i.e. the value of a given parameter for each pixel in the raster. This raster maybe either a change raster or an index raster. We de ne these notions in the next section. The goal of the process is to compute ROIs from this raster by aggregating "close" pixels for which the parameter has a minimum value. The process outputs an RDF KG representing these ROIs according to a formal vocabulary. The overall process can be described in 4 steps presented in Section 2.3) and detailed in the following sections. 2.1
We consider two kinds of rasters generated from EO images: index rasters, computed from a single image and change rasters, obtained by comparing two images. An index raster results from an image analysis at the pixel level. Several dedicated algorithms are available according to the phenomenon to be identi ed, like nature of the land cover, probability of re or water ood. For each image pixel, they compute some of its spectral values to generate a numeral value for this index. For instance, the algorithm that computes the Burned Area Index (BAI) produces a raster with values between -1 and 6. Several index rasters are available as open data like Corine Land Cover. A change raster (Figure 1 (b)) is produced by a change detection algorithm (as further developed in Section 3) from a couple of satellite images having the same footprint, and captured at di erent dates. The footprint and the dates are part of the raster metadata. Its content is a grid in which each pixel is associated a change value according to the change level computed by the algorithm. In our use cases, changes are identi ed by an unsupervised machine learning algorithm. Each pixel has a value between 0 (low probability of change) and 1 (high probability of change). 2.2
In image processing a Region Of Interest (ROI) is a subset of an image identi ed for a particular purpose. In the case of EO images, ROIs give some information about an area on Earth (the footprint). Thus an ROI can be geolocated thanks to a polygon included in the raster area. ROIs can be automatically identi ed thanks to the raster le analysis. In this case, criteria on pixel values determine what is an area of interest. In this paper, ROIs are not calculated from an image raster, but rather from a raster le that provides some information at pixel level for the same area and at the same date. Such raster les are the above-mentioned change rasters or index rasters with a link towards the image raster(s) that they are computed from. More precisely, we de ne an ROI as a subset of a raster le selected according to two criteria : i) most of the pixels in this subset have a value higher than a minimum threshold (which reveals something to be noticed); and ii) the polygon of adjacent high level pixels has a minimum size. To speed up the computation and the search for ROI, we decided to represent them by rectangles corresponding to their bounding box. Practically, the ROI is the bounding box including several close rectangles that gather pixels from a raster le with a value higher than a threshold value. Each region includes spots of interest rather than delimiting their exact perimeter. 2.3
The KG generation process takes a change raster and the various contextual datasets as input, and generates a graph of RDF triples. The notion of ROI plays a major role in this process as a means to aggregate information available at the pixel level inside continuous larger regions. This notion is computed in a bottom-up way for each parameter or index using the values of this parameter for each pixel. The process starts with the selection of the kind of event to be studied, and the knowledge required to characterize this type of event. A list of indices and several sets of contextual data are selected for their relevance. For instance, in the validation use case which focuses on wild res (Section 3), we selected the following types of information as relevant to identify and locate a re: indices like Change Area Index (ChAI) and Burned Area Index (BAI), location names and tweets as a way to con rm that a re happened. We get this information from two kinds of data sources: the EO images themselves, from which indices are calculated thanks to speci c algorithms, and open data sources. We give here an overview of the main steps of the process. We explain how ROIs are computed and used for tripli cation in the next sections. 0. Raster selection We select a raster le according to the information it conveys: either change data or index data. Among the raster metadata, we will exploit the date, footprint and the links towards one or 2 images from which it was generated. 1. Identifying regions of interest Given a raster le, we extract the image footprint (its geometry) and the raster period. Then, parsing the raster, we look for ROIs thanks to two parameters: i) the minimum threshold to consider a pixel value; and ii) the minimum area of a polygon of adjacent high level pixels. 2. Retrieving contextual data Considering the footprint geometry and the change period, contextual data with spatial and temporal properties is retrieved from di erent data sources and in various formats (shape le, WKT, CSV, ...): open data, open Linked Data, (i.e. or private data. 3. Generating the KG A set of transformation rules is applied to extract the appropriate data from each selected source and to create an RDF graph. These RDF graphs are conceptually linked together into one KG, as soon as a node identi er (URI) appears in each of them.
The resulting KG can be used to retrieve the number of ROIs a ected by a phenomenon and their size. In a wider scope, the KG can also be used to tackle more speci c downstream analysis tasks: visualization of feature changes over time, together with the corresponding satellite images; exploration of supplementary information related to the ROI using additional knowledge bases that we link to the graph for further extending the analysis indicators; etc. 2.4
To generate ROIs, we developed Algorithm 2.4, which takes a raster le as input, and produces a shape le containing all polygons with highest values, and then computes their envelopes as bounding-boxes.
Polygonizing change areas. From a change raster our algorithm produces rst (line 1) a new raster with binary values (0 and 1) where the value 1 characterizes a pixel having a change value above a given threshold. A polygonal representations of each change area is then computed (line 2).
Computing ROI envelopes. From this list of polygons, those with an area below a threshold (provided by minArea) are rst excluded (lines 3 - 8). A step of simpli cation and aggregation of close polygons is repeated as long as the list of polygons cannot be reduced any further by number. In fact each execution of the loop (lines 10 - 14) simpli es the list further. The function Envelope() (line 12) simpli es the shape of each polygon given their geometry and returns the minimum bounding box of each of them. The function CascadedUnion() (line 13) joins polygons that intersect and returns their bounding boxes (their geometry). At the end of the process all resulting polygons are disjoint and have an area superior to the minimum area (the minArea parameter) de ned to consider a polygon as a ROI. Figure 2 shows the stages and results of the simpli cation and aggregation process applied on the top left part of the raster of the Figure 1(b) after 6 steps: (a) represents the output of the Polygonize function, i.e. a shape le that contains thousands of small disjoints polygons; (b) to (g) are the results of each iteration that takes the envelope of each polygon and joins the ones that overlap; (g) is the nal result where no more simpli cation can be applied. Algorithm 1 The ROI generation from a raster 1: highIndexRaster F ilterHighIndex(raster; threshold) 2: listP olygons P olygonize(highIndexRaster) 3: listROIs ; 4: for each polygon in listP olygons do 5: if polygon:area >= minArea then 6: listROIs:add(polygon) 7: end if 8: end for 9: nbP olygons 0 10: while len(listROIs) 6= nbP olygons do 11: nbP olygons len(listROIs) 12: listROIs Envelope(listROIs) 13: listROIs CascadedU nion(listROIs) 14: end while 15: return listROIs (a) (b) (c) (d) (e) (f) (g)
Fig. 2: Evolution of ROIs per iteration during the process of simpli cation. 2.5
To integrate this data, like [23], we rely on the spatial dimension of entities. to
reduce the cost of computing spatial relationships between entities representing
data and images, in the sense of [22]. However, we chose to limit ourselves to
the spatial relationships de ned by GeoSPARQL in order to use this language
to query the data. In a second step, the integration takes into account the
temporal and spatial properties of the data, as detailed in the following. The KG
generation (Figure 3) is based on a vocabulary we have de ned reusing
standard vocabularies, namely GeoSPARQL [
Fig. 3: Extract of the vocabulary used to describe the mapping templates.
We evaluate the quality of ROI generation and RDF representation in the scope
of re detection from EO images. Because computing the change raster is still
quite time consuming, we start with a qualitative evaluation on two images,
which provide two use-cases. The rst one is the Kincade Fire (11/2019, Figure
1), whereas the second one is the Camp Fire introduced in [
Generated KG A KG is generated from ROIs as described in 2.5. It is also enhanced using the other data sources. Administrative units and re points are instances of geo:Feature. The spatial dimension of a tweet keyword is the geometry of the administrative unit associated with it. Tweet keywords and re points are linked to time:TemporalEntity instances.
Setting Change ROI thresholds To validate the Change ROI generation algorithm (Algorithm 2.4), we set up optimal values for two thresholds. Both of them are used to reduce the number of generated ROIs on a raster. The rst parameter, the threshold pixel value, is used by the "polygonize" algorithm as the minimum value for a pixel to be selected in a polygon. It is used to generate a mask that lters out pixels that form polygons. The second parameter to be con gured is the minimum area of a polygon in square meters. It plays the role of a second lter to select, among all the polygons proposed by polygonize, the one to be considered as a relevant ROI according to the type of event to be studied. The smaller the size, the larger the number of output polygons, many of them being not relevant. On the contrary, choosing a too high value may lead to silence: less polygons are generated, so few that some important ones may be missing. We empirically observed that a pixel value threshold below 0.65 led to noise whereas a threshold higher than 0.65 generates too few polygons, leading us to set this parameter to 0.65. Then we can choose a minimum polygon size
4 http://yago2geo.di.uoa.gr/ 5 Tweets are extracted from a collection captured at IRIT since 2015 and stored on the OSIRIM platform https://osirim.irit.fr/site/fr/articles/corpus between 5; 000 m2 and 15; 000 m2 to generate a reasonable amount of polygons. We selected 10; 000 m2 as minimum area value of a polygon because 1 hectare is a reference unit in remote sensing and especially for the study of forest res. Evaluation of Change ROI generation. To evaluate the use-case output polygons we use a shape le provided by the California government. This shape le represents the exact perimeter of the Kincade re. We aim at demonstrating that the calculated change ROI covers most of the re surface. In Figure 4(a) we can see that almost the entire perimeter of the Kincade re is contained in the main ROI polygon generated by our algorithm. According to the CalFire website, the Kincade re area was 31468ha and the generated ROI that covers this shape is 70905ha. Thanks to the area we are able to identify where the most important change is located. ROIs allows us to interpret a change without seeing the satellite image nor the change raster.
change ROI. Generating BAI polygons aims at extracting from the image all highly burnt areas, i.e. the areas having a high burn index. To do so, we set the threshold value of the burn index to 1 (on a scale from -1 to 6). Each polygon gives us the location of a previous re thanks to its bounding box. The polygonize function produces much less polygons on BAI rasters than on change rasters. However we applied the same threshold value for the minimum polygon area: 10; 000 m2 (1ha). As we can see in Figure 4(b) that most of the Kincade re perimeter is included in the generated BAI polygon.
Evaluation of the data query. The query available online6 retrieves all the ROIs, the burnt area in the ROI, the number of linked re points (from FireCast) and the keywords about the cities in the ROI for the period. From the result of this query, we can get the ROI that contains the highest number of re points in the area of this raster and the keywords associated to the administrative unit that intersects with this ROI. Keywords can contribute to determine if there is a re in the ROI. In Figure 5 (a), we can see that a ROI intersects with two administrative units. The word \ re" is one of the keywords detected in tweets at this period for the city of Healdsburg in California. Hence, contextual data explains that the change observed on this area was caused by a re. Data from tweets properly documents the kind of event that caused the detected change.
Evaluation on a second use case. For the Camp Fire which took place in California in November 2018 7 the authors detect the forest re (a burn map) using change detection over a CNN-based framework without contextual data. Using the same parameters as the one de ned for the rst use case, we obtained similar results. As shown in Figure 5 (b) we successfully found keywords \ re" and \wild re" for the concerned administrative units. This use-case enabled us to use Landsat 8 images as input for the change detection algorithm. Even if the spatial resolution is lower we retrieved the same quantity of data.
contribute to validate the results from the machine learning algorithm. To do so, we cross-checked the results obtained from the di erent sources on the same location and period of time as in the previous section. The results from deep learning process used for change detection show a high change on the image. The Firecast data also show many re points in the area that explain that the change is due to a re. This information is also corroborated by the data collected from tweets, that report a forest re called Kincade Fire 8 in this area of California during this period. Contrariwise, one can observe a re in a smaller area in the lower right part of the image, as indicated by a grouping of re points. However, this re has not been detected as a signi cant change from the change raster because there are not enough pixels with a value above 0.66 to detect an abnormal change. Nevertheless, our processing chain was able to detect multiple re points thanks to the Firecast data (Figure 4). In that way, our approach is able to identify false negative areas of the change. Moreover, it corroborated the changes detected by the unsupervised algorithm. In a further work, RDF data could be more tightly coupled with the change learning process to improve it results, for instance by providing feedback or training examples.
Annotating the images with external data or with image indices has been
addressed in di erent studies. In [18], vegetation indices result from satellite image
processing and are exposed as RDF triples using GeoSPARQL [
This paper proposed a process for generating a KG for the semantic quali cation of automatically computed changes between two images. This process is guided by the notion of Regions of Interest (ROI), as an accurate geographic division representing a high probability of changes. This can be a basic brick in a more complex process of monitoring events on the Earth's surface, as it is able to automatically identify the location, nature, beginning and end of events. We plan to carry out new experiments to con rm the generality of the integration process, using similar or new types of events (i.e. urbanization, migration ows, ...) and other types of contextual data. We will also investigate how to reuse the RDF data to improve the change detection process. Another task is to improve the keyword selection using a Tf.Idf algorithm to get more relevant keywords and to validate these words against a lexical eld of natural disasters. A second perspective would be the use of trained neural models as additional knowledge sources when extracting keywords.
Acknowledgements This work bene ted from CANDELA H2020 grant (convention number 776193); and ANRT CIFRE PhD. scholar (number 2017/1399). 16. K. R. Kurte and S. S. Durbha. Spatio-temporal ontology for change analysis of ood a ected areas using remote sensing images. In FOIS Workshop, 2016. 17. K. Kyzirakos, M. Karpathiotakis, and G. et. al. Wild re monitoring using satellite images, ontologies and linked geospatial data. Web Semantics, 24, 01 2014. 18. A. J. Luque, R. Perez, F. J. Bonet, and P. Magan~a. An ontological system based on modis images to assess ecosystem functioning of natura 2000. Journal Applied Earth Observation and Geoinformation, 37:142{151, 05 2015. 19. N. Pittaras, G. Papadakis, and e. a. Stamoulis. Geosensor: Semantifying change and event detection over big data. In SAC, pages 2259{2266, 2019. 20. R. J. Radke, S. Andra, O. Al-Kofahi, and B. Roysam. Image change detection algorithms: a systematic survey. IEEE Trans. Image Proc., 14(3):294{307, 2005. 21. B. Shbita, C. A. Knoblock, W. Duan, and et. al. Building linked spatio-temporal data from vectorized historical maps. In Semantic Web, pages 409{426, 2020. 22. M. A. Sherif, K. Dre ler, P. Smeros, and A. N. Ngomo. Radon - rapid discovery of topological relations. In AAAI, pages 175{181, 2017. 23. P. Smeros and M. Koubarakis. Discovering spatial and temporal links among RDF data. In Workshop on Linked Data on the Web, 2016.