Scientific efforts to observe the state of natural systems over time, allowing the prediction of future states, have led to a burgeoning interest for organised storage of spectral field data and associated metadata, seen as being key to the successful and efficient modeling of such systems. A centralised system for such data established for the Australian remote sensing community aims to standardise storage parameters and metadata thus fostering best practice protocols and collaborative research. Supported by the Australian National Data Service (ANDS), whose aim is to promote connections between data, projects, researchers and institutions, a spectral information system based on the already operational SPECCHIO spectral database system is being augmented to specifically meet the needs of the Australian remote sensing community, and is aligned with the Terrestrial Ecosystem Research Network (TERN) Auscover facility. In this paper we outline the envisaged dataflow and usage of the system as a case study within the context of TERN Auscover. The development of a national spectral information system will not only ensure the long-term storage of data but support scientists in data analysis activities, essentially leading to improved repeatability of results, superior reprocessing capabilities, and promotion of best practice. Andreas Hueni's main research interests are the development of spectral data processing systems and databases, and the calibration of imaging spectrometers. He is the author of the SPECCHIO spectral database and APEX (Airborne Prism Experiment) calibration and processing scientist. Laurie Chisholm's research interests focus on the application of advanced spatial science techniques and the development of spectral indices to investigate the impacts of disturbances on ecological systems at a range of scales. She is involved in the compilation of spectral libraries in support of research and validation of image products and manager of the ANDS Project DC-10, development of a national spectral database system. Lola Suárez holds a PhD in remote sensing of vegetation stress. Her research interests include vegetation biophysical properties estimation and up-scaling methodologies to different spatial resolutions using hyper/multispectral remote sensing, and radiative transfer modeling. Cindy Ong's research interests are focused on deriving quantitative measurements from hyperspectral data. She has extensive experience on the application of hyperspectral technology for environmental measurements for the mineral industry and regulatory requirements. Mathew is a technical manager and analyst on a number of eResearch projects around Australia, and avid supporter of open and collaborative data projects such as the DC-10 project.
Field spectroscopy, particularly the measurement of the spectral reflectance of terrestrial or aquatic features, is widely
used as a tool for a variety of research operations and applications and plays a critical role in the calibration and validation
of image data from remote sensing campaigns
In Australia there is a wealth of field spectroscopy data collected from a variety of instruments providing measurements at multiple scales and for wide-ranging purposes: as ground truth collected in support of a remote sensing validation campaign to relate image data to real features and materials on the ground; to calibrate remote sensing data and sensors; to provide a spectral reference for materials on the Earth surface; to determine mixtures of materials or investigate discriminatory capacity to discern a particular material or examine the behaviour of spectra from influencing factors; and to answer research questions across a multitude of disciplines.
Data from these wide-ranging capture events are often stored in silos, for example, on ftp stores, local machines, in data disks. Historically such data are not well documented. These data, especially if independent validation data were collected concurrently, are extremely valuable yet are not discoverable by others doing experiments in similar areas, on similar sample types or requiring references for certain materials. There are also generally no linkages between the remotelysensed data, the field validation spectra and the independent validation data for example XRD (X-ray diffraction) data for mineralogy; vegetation moisture content for bushfire fuel assessment.
Metadata are paramount to broad and long-term use and interpretation of scientific data and must thus be acquired and
stored in a rigorous way
Spectral databases are systems for the storage of spectral data acquired by spectroradiometers under both field and
laboratory conditions, augmented with associated auxiliary data
Most essentially, spectral information systems are no longer mere data repositories but systems that support the scientists
in analyzing their data utilizing the full potential of combined metadata spaces
The aim of this project is to develop a centralised system to house spectral libraries and associated metadata. The
repository will allow the Australian remote sensing community to collate, share and discover existing spectral libraries
and facilitate the capture of new datasets as they are formed.
The system specifications presented hereafter are based on a requirements and gap analysis carried out during a TERN
ACEAS workshop on best practices associated with bio-optical data in mid-2012. Involving stakeholders from multiple
disciplines within the Australian and international remote sensing community, the gap analysis has been expanded with
an aim to consolidate and prioritise system requirements to ensure the final product meets the needs of end users. Many
required features are already provided by the current SPECCHIO system
Insert of new data records into existing collections Delete data from database Granularity of metadata description
SPECCHIO data remover function
Metadata are stored at single spectrum
level, but group updates are possible to
streamline input processes
Via SPECCHIO policy concept and
SPECCHIO metadata space density
measure
Via metadata space restrictions defined in
the SPECCHIO query builder
Data can be owned by a group instead of only a single user A data set is also owned by a PI (principal investigator) Augmenting spectral collections with metadata held by files Add new file reading routines as interfaces to such systems Metadata editing action logs with reference to involved user (provenance information) Use concepts developed at RMIT, Melbourne User group specific data editing masks, based on sampling protocols Freeze data at a given state and disable editing Educational, training, test, research, etc.
Defined on campaign level.
Adding new persons to an existing group for a limited time, e.g. for reviewing Facebook similar features Data can be accessed without local installation of SPECCHIO desktop application Some data can be embargoed by the PI for a certain time while work or publishing is ongoing Store a copy of the files on a fileserver during data Export of data to e.g. GIS formats ingestion and make them available for download Add exporting routines for identified software required by end users
An initial gap analysis identified the potential of the SPECCHIO system to serve as a basis for the development of a national spectral database for the Australian remote sensing community. The following methods thus represent a mix of the current capabilities of the SPECCHIO system and of features that will be developed or enhanced in the scope of the spectral database development.
The general dataflow (Figure 33) starts with a data ingestion process, loading spectroradiometer generated data files to the database. This step involves extracting as many metaparameters as possible from the files to reduce the manual metadata input. Input files supported at the time of writing are given in the appendix.
Metadata of data contained in the database can be augmented by using a metadata editing graphical user interface (GUI). This process is streamlined by the concept of group updates, where a spectral collection can be updated in one operation. Data held by the system can be processed and visualised within the SPECCHIO Java application, but these capabilities are rather generic and rudimentary as it is virtually impossible to cater for all possible user needs within the Java application. Data may be exported to regular files with CSV (comma separated values) and ENVI spectral library being the current output options.
More complex or user specific processing, analysis and visualization can be implemented in programming languages that
support scientific, exploratory work at a much greater extent than Java. Such higher-level algorithms are easily
implemented in any language that supports Java bridging in some fashion. Generally, SPECCHIO Java Objects are used to
connect to the database, making spectral data and metadata available to the runtime environment of choice. Matlab and
Python have been proven to seamlessly interact with SPECCHIO databases and the same is expected to hold true for
further languages such as R and IDL. APIs (Application Programmer Interfaces) provided by the SPECCHIO system
software support all interactions with the relational database management system (RDBMS), i.e. querying, inserting,
updating and deleting (Figure 2).
This capability is extremely powerful, as it gives scientists the freedom to develop their own algorithms and data
analysis/processing flows while basing on a stable and abstracted foundation that handles all the querying, data loading,
inserting as well as housekeeping regarding metadata and consistency of spectral spaces.
The SPECCHIO system can be easily deployed on database servers with or without Internet access or on local workstations
or even on field laptops. A possible ontology is shown in Figure 44. This allows a flexible and user-need tailored use of the
system, e.g. to limit the data access to in-house researchers, or to facilitate data storage and processing during field
campaigns where Internet access may be impossible. SPECCHIO offers a file-based option to exchange data between
distributed databases and thus allows consolidating the data in a central server once deemed appropriate to do so by the
data producer
The Terrestrial Ecosystem Research Network (TERN) AusCover facility was created to facilitate the production of validated satellite-derived biophysical map products for the ecosystem research community and natural resource managers (TERN 2010). The Auscover network strives to create these products within the framework of established Australian validation programs following established international earth observation protocols.
Using a TERN Auscover field campaign as a case study illustrates the use of the current SPECCHIO system as data repository and as a platform for post-processing and storage of according results in the database. This study was chosen to demonstrate the generic nature of the SPECCHIO system for the handling of in-situ spectral data and metadata and thus provides a preview of the future system functionality.
In April 2012 hyperspectral and Lidar imagery were acquired over Rushworth State Forest in Victoria. The study site consists of a reference area located within the main box-ironbark forest 19 km NW from Nagambie. Simultaneously, field data collection was conducted consisting of vegetation structure measurements and leaf sampling. The aim of this field campaign was to characterise leaf chemistry and spectroscopy for a number of trees in the study site. A total of 96 stands representative of the five most abundant species were selected. Only trees which crown spectra could be extracted from the imagery were considered for selection. As a result, none of the selected trees were suppressed or shadowed by the surrounding crowns. From each selected tree, one branch from the upper-most third of the crown was shot down and a set of mature leaves was taken for further analysis. Additional metadata collected for every tree consisted of stand coordinates, tree species, height, trunk diameter at breast height (DBH), crown diameter, crown percentage cover and position of the crown in relation to the surrounding trees (i.e. dominant, co-dominant or isolated). In the following hours, an integrating sphere (Analytical Spectral Devices, Boulder, CO) was used to measure reflectance and transmittance spectra of three of the leaves collected per tree. Another group of leaves were scanned and weighed to estimate specific leaf area, and a third set was sent to a laboratory for chemical analysis.
The spectral measurements were acquired using a FieldSpec Pro spectrometer (Analytical Spectral Devices, Boulder, CO) attached to the integrating sphere through a bare fiber. Measurements to correct for the misalignment of the light source were taken for every reflectance and transmittance measurement.
During data ingestion the SPECCHIO ASD file format reader loads data from the file system into the database. This is a standard process and data are immediately available for sphere post-processing. To give users a better control of the processing applied, an interactive graphical user interface written in Matlab using SPECCHIO Java components is used (Figure 66). The left hand side of the GUI holds the Spectral Data Browser component, displaying the content of the database in hierarchical form and allowing the selection of spectral data. Such a selection is the starting point for the dataflow ensuing as illustrated in Figure 5. Once data are selected, their database identifiers will be loaded. These are then iterated over during the spectra loading, selecting spectrum records from the database and making the spectral data available as Matlab matrices. Two modes are supported by the GUI: (a) group by group interactive processing and (b) automatic processing. The former option processes one group consisting of the four raw measurements, displays the raw spectra and the resulting corrected reflectance and transmission spectra in the six dedicated display panels of the GUI. The automated mode processes all groups in one operation and displays all corrected reflectance and transmission spectra; no raw spectra are displayed in that case.
These corrected spectra are then augmented with metadata and inserted into the database, essentially copying the metadata of the input spectra and changing them to reflect the processed state, i.e. setting the correct measurement type of either reflectance or transmittance.
Implementing such a tool can be accomplished in a matter of hours by basing on SPECCHIO API’s, resulting in most of the development effort being concerned with data handling and GUI representations within Matlab.
The case study proved that the current system is very well capable of storing and processing data for a specific use case in an operational manner by relying on generic system functions. In particular, it could be shown that (a) data can be automatically ingested, (b) data may be retrieved from the system using metadata queries in a sequence dictated by the scientific experimental setup, (c) a use case specific higher-level interactive component could be easily developed, and (d) extracted information could be re-inserted into the database, automatically linking with the existing metadata of the input spectral information. The system thus greatly increases the speed of data processing while reducing operator errors and linking resulting information to existing metadata, thus ensuring retrievability. As outlined, the development of a national spectral information system for the Australian remote sensing community, aims to standardize spectral storage parameters and metadata, thus promoting best practice protocols. By enlisting the input of remote sensing stakeholders across wide-ranging disciplines, it is anticipated that the system developed will provide tools that researcher’s need and desire, thus enhancing current workflows and facilitating collaborative research. The project, starting in the last quarter of 2012 with an expected duration of 5-6 months, aligns with ANDS objectives by creating descriptions of the spectral libraries and sharing those descriptions in ARDC (Australian Research Data Commons). Once developed, there will be an automatic feed from the system to ARDC. Data from the libraries will also be available for reuse by other researchers.
The software deliverables will be of use to other research groups who also have spectral databases and its availability will ensure any future libraries are built on spec based on protocols and metadata requirements, thus, facilitating use and growth of the system. Since libraries can be built at multiple scales and levels, the potential use of this development offers high value return on investment, based on a common need across the entire remote sensing community. In addition, the system will enable reliable transfer of experimental data for calibration and research and promote sharing amongst collaborators and other researchers via ARDC.
Interested users may test the current SPECCHIO system at any time by subscribing to the online SPECCHIO spectral database via the SPECCHIO website www.specchio.ch.