Many applications of cyberphysical systems rely on the largescale integration of geospatial data and sensor data. These types of data are both characterised by their heterogeneous structure, di erent formats for their representation, and their enormous volume. In recent years, the Semantic Web community has developed vocabularies and standards for representing and querying geospatial and sensor data. In this paper, we present an ongoing research project addressing the need for a exible and scalable platform of services handling such data. The platform comprises services for the extraction, transformation, interlinking, fusion, quality assurance, and delivery of smart data. In contrast to related work, we pursue a message-driven approach for a exible and scalable integration of microservices. We further discuss several use cases which motivate the design of our system architecture.
An increasing number of applications depend on the exible integration of live data, such as sensor data. Most sensor data is inherently related to geospatial data, e.g., the location of a certain measurement or place references in metadata. Other sources, like social media feeds, can contain place names that are valuable for real-time applications.
Despite the advances in research and technologies for the semantic integration of geospatial data, it is still challenging to build respective applications. It is particularly di cult to utilise live streams of data with geospatial aspects, such as sensor data, in real-time applications.
The di culty is mostly caused by three aspects. First, components for translating data into RDF, analysis, interlinking and enrichment have to work with typically small but frequent payloads of data streams. Second, the communication between these components needs to be exible and reliable. Third, the entire system has to scale dynamically according to a changing load of messages, which typically requires a distributed implementation.
Examples of respective applications include routing, geomarketing, and all sorts of analytics, such as business intelligence and predictive maintenance. Internal case studies for the use cases discussed in Section 2.1 indicate that these can be seen as a work ow based on reusable components for di erent process phases. Various data streams are provided as input. The data of interest is extracted and transformed into a common format, interlinked and fused with related data. The output might be examined with quality metrics before it gets stored and visualised.
There are many tools available already for handling these general tasks with Semantic Web technologies. However, hardly any of them support the speci c needs of real-time geospatial data streams. Also, there is no framework that provides a runtime environment for the integration of such tools and addressing additional requirements.
In this paper, we present an ongoing research project focusing on the following primary research question: How to design a platform for exible integration of di erent geospatial and sensor data (Section 2)? We propose a microservicebased message-driven approach, validate it (Section 3) and contrast it to related work (Section 4). 2
Based on a range of use cases and corresponding requirements, we derive objectives and an architecture for the GEISER project. Then, we describe the current implementation of the GEISER platform. 2.1
Use Cases We develop and evaluate the GEISER platform along the following three primary use cases.
Data-driven geomarketing. With more than 500 billion euros in retail revenue in Germany alone, 3 local retail and catering are important employers and taxpayers. To be successful in a competitive market, regional companies have to adapt their products, services, marketing strategy, opening hours and special o ers to their audience. Data-driven geomarketing would already be possible today using all available sources: the Web, social media and news, message boards, mobility and cellular network data, open data, their own customer data, etc. However, these sources are monolithic and lack explicit geospatial references.
In GEISER, we want to build a geomarketing assistant application which informs companies in near real-time considering all relevant information. For 3 Source: https://de.statista.com/themen/136/einzelhandel-in-deutschland/
example, a local restaurant would be alerted about a band playing nearby, extracted from social media and event data (e.g., Twitter), so it could provide special o ers to their fans. Further requirements include predictions based on weather warnings and forecasts (data from DWD.de and openweathermap.org) and public transport (via data.deutschebahn.com).
Intelligent parking. Roadside parking is challenging in crowded city centres. While personal navigation assistants (PNA) can guide users to a certain destination, they are not aware of available parking spaces on that road. Existing solutions to that problem require infrastructure support, so they only work in locations where sensors installed in the road already identify cars.
In GEISER, the messages of connected PNA are used to detect where cars have started and nished their journeys. Based on that data, it is possible to determine the probability for nding a free parking space on a given section of a road at a certain time. Instead of driving to a destination with a low chance of nding a parking space, the PNA would provide approximate guidance through nearby roads with a higher probability.
This probabilistic routing can be enhanced with more detailed information on how many people frequent a region at a certain point in time, and why they do so. Therefore, we need to interlink and interpret di erent data sources, including the weather data discussed above, local events (from Twitter) as well as oating car data and tra c incidents from TomTom. We build an extended parking search service based on the GEISER platform and On Street Parking. 4 Predictive maintenance and industrial service. Industrial service is becoming increasingly important. Sensor networks in industrial appliances generate huge amounts of data that can be used to predict a device failure based on historical information. Thus, industrial service can be planned and executed even before a failure occurs, reducing the cost of a halted manufacturing process.
In GEISER, we plan to link the predictive maintenance with service technician scheduling and spare part logistics. For an optimal service coverage, many other data sources with geospatial aspects have to be linked to industrial sensor data. Goals include a less unnecessary travelling and improved industrial service quality. Requirements include the integration of service technician deployment planning5 with weather and tra c data sources discussed in the other use cases. 2.2
Goal and Objective As shown in the use cases above, there are many applications that require exible processing of data streams from di erent sources with geospatial or temporal references. However, these relations are rarely inherent to the data or otherwise available in a machine-processable way. 4 https://developer.tomtom.com/on-street-parking/documentation 5 Data coming from the STEP project: https://www.projekt-step.de/en/
GEISER Services
Compatible Tooling (loosely coupled) e c n a n e t n i a M & t n e m y o l p e D trcy i u e S & n toi a c tni e h t u A n toi a r u fgni o C e c i rv e S & m r o tf a l P
K D S & s e c ti rc a P ts e B . v e D
Application Services (only samples included, use case specific)
Generic Services,GeGe.gne.neLreIirMcicSEeSSre,vrDivcEiecEseRs, FOX, AGDISTIS Data Services (querGyGeinnegne,remirciacSneSiprevurivlcaiectiseosn, geospatial, etc.)
GEISER Platform
A M Q P B U
S
Private or public cloud (Semantic Databases, SPARK ecosystem, SANSA …)
Therefore, it is the primary goal of the GEISER platform to enable mass processing of geospatial and sensor data with the following aspects: 1. Data extraction recognising the references between data and known entities 2. Semantic technologies for adding these references to the data 3. Flexible microservice architecture and asynchronous messaging for a scalable foundation of di erent applications
The GEISER platform is comprised of several horizontal and vertical layers, as shown in Figure 1.
Three horizontal layers divide the lower and higher level services available in the platform. The lower layer consists of data access and handling services, such as extraction services and triple store access. The middle layer contains generic platform services for processing semantic data, such as quality assessment, interlinking, and fusion services. The upper layer contains usually domain-speci c higher level application services. Examples include the probabilistic routing service for the parking use case and a target audience estimation service for geomarketing.
All of these services can access each other via a message bus. We chose the RabbitMQ implementation of the AMQP standard. This enables di erent message exchange patterns, including reliable asynchronous messaging with delivery guarantee. In addition to the exibility of the message routing, which can easily
Features: visualization, exploration, dashboarding, data analytics, semantic search, etc.
Examples: metaphactory, Facete, mappify, etc.
be modi ed at runtime, this brokered approach was favored to a much more
limited, entirely REST-based implementation. We also favored RabbitMQ over
Apache Kafka, primarily because of simpler consumer implementation, support
for the MQTT protocol supported by many sensors, and wider support for
different messaging patterns such as RPC [
The vertical layers provide the platform support for cross-cutting concerns like authentication, deployment, and service con guration. Application access is enabled by a REST API on top of the service layers. This allows external applications, such as the routing function in a PNA, to invoke services in the platform and receive results without having to be a message bus endpoint. Additional tools provide further functionality, including visualisation and semantic search. 2.4
Implementation Since the project is still in an early phase, we have only started to implement and validate our design decisions.
In contrast to existing semantic service infrastructures, which typically utilize HTTP requests for service interactions, we have implemented an asynchronous messaging approach. An AMQP topic exchange can be applied very exibly to implement di erent messaging patterns, including publish/subscribe-like interactions. These can also be modi ed at runtime, enabling ad-hoc changes to the processing of messages.
Additionally, GEISER proposes a message passing approach between a linear chain of services. For each use case, messages ow through a list of components. In order to specify the respective services, we store their topic identi ers in an AMQP header eld. This is usually con gured for the service sending an original event to the message bus. The remaining services are encoded as service1.service2.{...}.service n in the AMQP routing key. Consequently, each service is aware of the intended following routing. The following example shows a JSON-LD message from an interaction in a chain of services. The FOX service has performed Named Entity Recognition (NER) on the given text and sends the results to the following service location which annotates the geocoordinates of the entity.
Listing 1. Message processed by the entity recognition service fox, to be processed by further location and store services. Note: lines starting with two slashes are basic properties of an AMQP message, not part of the JSON-LD message content // routing key : location . store // message - id : 2 // correlation - id : 123 // app - id : fox // reply - to : an not at ion // content - type : a p p l i c a t i o n / ld + json // content - encoding : utf -8 // type : L o c a t i o n R e q u e s t // header :
// [ services : [" annotation ", " fox ", " location ", " store "]] { " @context ": " http :// example . org / context / fox . jsonld ", " text ": " Leipzig is a beautiful city ." , " fox :": [{ uri : " http :// dbpedia . org / resource / Leipzig ", name : " Leipzig " }] } Many services are already available with a REST API, but don’t feature AMQP support. In order to provide a simple option for integrating them, we have developed a wrapper approach for translating AMQP messages into REST requests and vice versa, based on a declarative service description. 6 The current implementation is based on NodeJS and Go.
Furthermore, we implemented a small library based on Spring AMQP for implementing the message bus interfaces for Java components. The following listing shows that very little code has to be implemented for a GEISER message handler. The handleRequest method is invoked with the unmarshalled request and the original AMQP message. Besides the @RabbitListener annotation specifying the topic this service is listening to, the library provides request (un-)marshalling and extracting the next routing key for the response message.
Listing 2. Stub for implementing a GEISER service interface in Spring Boot /* imports */ @Component public class MyService { public static final String ROUTING_KEY = " myservice - v1 .#"; public static final String QUEUE_NAME = " myservice - v1 "; public static final String EXCHANGE = " geiser "; @RabbitListener ( bindings = @QueueBinding ( key =
ROUTING_KEY , exchange = @Exchange ( type = ExchangeTypes . TOPIC , value = EXCHANGE , durable = " true ", autoDelete = " true ") , value = @Queue ( autoDelete = " true ", value = QUEUE_NAME ))) public void handleRequest ( Request request , @Payload
Message message ) throws IOException { /* handle request */ String nextRoutingKey =
ServiceUtils . nextRoutingKey ( message ); rabbitTemplate . send ( EXCHANGE , nextRoutingKey ,
MessageBuilder ... build () ); }
} 6 https://github.com/AKSW/micropipe-proxy
For monitoring an exchange we can listen to all topics of an exchange with a wildcard and store the messages, e.g., into a triple store. In contrast to storing the entire message as a literal, we decided to store each AMQP value as separate properties to be able to e ectively search that log using SPARQL. We realised that many properties that we had modelled for messages following a requirements analysis phase are already available as basic properties in AMQP. We only had to specify the details of respective values, e.g., how services are referenced in the app-id property representing the origin service of a message. Since to the best of our knowledge no such thing exists, we created a basic RDF vocabulary 7 for these AMQP properties.
For managing the deployment of an application consisting of services we use Docker and Docker Compose. Each service is dockerized and con gurable, e.g., using environment variables. The set of services required for an entire application work ow is con gured in a compose con guration le. 8 The following listing shows part of such a le for the geomarketing use case. When deployed using docker-compose up , the Twitter feed service will push new tweet messages mentioning Leipzig onto the message bus, which will subsequently be translated, annotated (by FOX) and stored.
Listing 3. Docker Compose con guration (partially) for a geomarketing use case services : geiser - twitter - feed : image : mwauer / geiser - twitter - feed links :
- rabbit : rabbit depends_on :
- rabbit environment : - keywords = leipzig - routing = translate . fox . store [...] translate : image : mwauer / geiser - translate - service links :
- rabbit : rabbit depends_on :
- rabbit environment :
- translate_input_attribute = sioc : content \[]\ @value rabbit : image : rabbitmq : management ports :
- 5672:5672 7 https://github.com/mwauer/geiser/tree/master/messaging-ontology 8 Further schema information: https://docs.docker.com/compose/compose-file/
So far we have implemented a set of generic services for reading and writing RDF, interlinking (using LIMES), NER and relation extraction (using FOX), enrichment and fusion (using DEER) and machine translation (using the Yandex API). Further custom services are being developed for the respective use cases, e.g., a probabilistic routing service.
While this implementation appears to be su cient for the presented use cases, we are currently evaluating whether more complex work ow patterns are necessary (see Section 3). However, these could easily be achieved with a microservice inspecting the incoming message and potentially modifying the routing key. Thus, conditional work ows can be implemented. 3
Due to the early stage of the implementation, we have focused on the functional evaluation of the platform and performance measurements of the components, according to the requirements gathered from the use cases. These are primarily related to data sources (various types of static and real-time data) and required services for processing and querying data.
We have successfully run an instance of the service composition shown in Listing 3, showing the applicability of the approach for a part of data-driven geomarketing use case described in Section 2.1. Further functional evaluation depends on the availability of further services speci c to the use cases, which are currently being developed.
With regards to the extraction of data from unstructured sources, the
evaluation of FOX in the OKE challenge [
Semantic Service Infrastructures
Since the beginnings of service oriented architectures, the use of explicit
semantics for integrating services has been investigated, e.g., by METEOR-S [
Semantic Automated Discovery and Integration (SADI) [
for simple service discovery and interaction, but it is no replacement for a directory service and lacks any support for asynchronous messaging or other features provided by a messaging infrastructure.
The Simple Semantic Web Architecture and Protocol (SSWAP) [
These approaches are relevant to the problem of a exible integration of services. However, they do not address the issue of asynchronous messaging.
Furthermore, there are approaches for machine-readable service descriptions
for REST APIs, such as MicroWSMO and hRESTS [
Semantic Data Processing In contrast to the above section, this part focuses on research on processing data which is described semantically, i.e., data that is represented in an RDF-related format.
LinkedPipes ETL 9 (formerly Uni edViews [
The GeoKnow project focused on semantic processing tools for geospatial
data. The GeoKnow Generator [
In addition to these relatively generic projects and frameworks, there are
ongoing research e orts relevant to the presented use cases. For instance, the
EW-Shopp [
In this paper we described the motivation and architecture of the GEISER platform. In comparison to related work, it will enable the scalable processing of geospatial and sensor data with semantic technologies.
Services for each Linked Data lifecycle phase, as well as lower-level generic infrastructure services and high-level use case speci c ones are connected via a message bus. The platform will also provide features for vertical concerns, such as non-functional requirements.
In future work, we plan to implement the use cases in detail using the
platform services. We will evaluate the exibility of the platform interfaces and
the scalability of the messaging middleware by benchmarking the system in the
HOBBIT platform [
Acknowledgement The authors would like to thank Michael Schmidt from metaphacts for creating the GEISER architecture gure.
The research and development project that forms the basis for this report is funded under project No. 01MD16014E (GEISER) within the scope of the Smart Services World technology programme run by the Federal Ministry for Economic A airs and Energy and is managed by the project management agency at the German Aerospace Center (DLR-PT). The authors are responsible for the contents of this publication.
Bibliography