Logistics is a complex industry where many different types of companies collaborate in order to transport containers to the last point. One of the most important problem in logistics domain is observation and monitoring of container life cycle where each step of the container transportation may be performed by different company. Thus, observing and monitoring of the container's life cycle in real time become a challenging engineering task. In this research, Linked Data development infrastructure has been used to implement dynamic container observation and monitoring system for ARKAS company which is the leading logistics company in Turkey. During the development of the system, it has been observed that agile practices like feature/story oriented development, test first development and usage of Agile Architecture approach improves the product and project management quality. So, a new methodology has been proposed based on these practices for Linked Data development.
Logistics is a complex industry where many different types of roles such as shipping agency company(ies), port/ship operator(s), land, rail and air transportation companies collaborate in order to transport a container to a destination. These companies require a collaboration infrastructure in order to resume the whole transportation process seamlessly. This collaboration infrastructure should provide an integration environment for the information systems of the companies in order to manage all sub-transportations and needs to be open in a sense that it should to be easy to be able to add/remove company(ies) into the process. In addition to the integration of the information systems, monitoring of the process is also critical for the effective management of the process. For instance, important events like delay in the completion of the land transport, starting of the discharging operation at the port can be very critical for the roles that are participated in the process.
This paper introduces an implemented architecture based on Linked Data infrastructure for the well known logistics problem “Observation and Monitoring of Container Life Cycle”. Application is developed within the ARKAS Holding which is one of Turkey’s leading logistics and transportations company. It operates in different fields such as sea, land, rail, air transportation, ship operations and port operations. Therefore, executing “Observation and Monitoring of Container Life Cycle” problem in real time is a challenging engineering task since all sub-transportations may run in parallel on different software systems of different companies. The end goal is to have managers and customers be able to track the container transportation life cycle.
In the last decade, EDI-based standards (EDIFACT, RosettaNet,
STEP, AnsiX12), XML standard and Service Oriented Architecture
(SOA) approaches are used for solving the integration problems of
logistics industry[
Linked Data infrastructure seems like an appropriate technical
solution for the requirements of the logistics industry, since it
provides an integration environment which is more flexible, extensible
and open to the exterior when necessary. Linked Data standards
and infrastructure are prevalently used to integrate enterprise
information and business processes[
During the development of the aforementioned “Observation and Monitoring of Container Life Cycle” application, the development team defined a development methodology. Since, the development team has a long time experience in Agile Development, the proposed methodology brings agile practices into the Linked Data development. The Methodology called as BLOB(A Methodology to Bring Agility into Linked Open Data Development for Businesses) has evolved through the iterations of the development. Its final version which is overviewed within the paper has following contributions:
At the moment, “Observation and Monitoring of Container Life Cycle” application is operational and tested by customer operation unit of ARKAS holding. The paper introduces how the application and its architecture evolved through the iterations of the proposed methodology.
In the logistics industry, customers’ loads are transported to a final destination from a start location within containers. Through this transportation, containers are proccessed in a variety of work areas such as port, warehouse, land, rail and air. For instance, let us consider a company that wants to send its load to customers in Munich/Germany from Manisa/Turkey. This company is aggreed with a forwarder company for the transportation. This transportation is planned as a four-stage; a land transportation from Manisa to Izmir Alsancak port, a maritime transportation from Izmir Alsancak port to Piraeus port of Athens, a maritime transportation from Pireaus to Venice port and a land transportation from Venice port to Munich. Also, it takes approximately 10 days and there is not an interface to get information about the transportation.
Throughout the transportation, customers want to learn exact
status and position of the transported loads and this problem is
dealed in wireless network and RFID studies[
Transportation history of the container which is distributed into the different software systems should be integrated.
Events of the container transportation in different work areas should be monitoring and customers should be informed about transportation status. 3.
It is well understood that agility is the good approach to cope with
changing business and architectural requirements in software
development. Not only software development, but also business
intelligence and analytics implementations benefit the agile style of
development as proposed in [
Linked Data development is very young domain where
critical tools and architectural patterns are constantly evolving. Also,
changes in business requirements and/or Linked Data environment
may affect the initial architectural assumptions. Thus, development
team should observe the evolution and the performance of the
architecture throughout the development. Observation of the
arhitecture evaluation throughout the methodology is clearly defined in
Agile Architecture approach [
Linked Data development requires some specific tasks as
defined in various Linked Data development methodologies[
In the analysis activity, application requirements are identified and
managed by product owner by incorporating with necessary project
stakeholders like customer role(s) and development team
member(s). As a first step of the analysis, main goals of the
applications are identified and for each main goal new user stories are
defined to satisfy the goal. The critical aspect of the analysis phase
from the linked data perspective is the identification of the data
sources that require for the user story. The data sources are
identified by development team and attached to the “Application
Requirement Card” (ARC). The second critical differences from the
classical user story definition is the competency question section
of the card. Competency questions is the well known approach in
the ontology development literature [
The ARCs are defined for each story including the following parts;
Selected Data sources the story: Data sources that will be converted to Linked Data and be consumed by the application. They can be relational databases, documents, web pages etc. User Stories: Senarios of use from the viewpoint of end-users in order to implement the defined feature of the goal.
Iteration Requirement Specification (IRS) is a planning activity. ARCs that are maintained in the ARC backlog are prioritized according to their business value and included data sources. Considering the data sources in story prioritization is critical in terms of Linked Data development. If more than one sources are included in the story, this situation affects the Linked Data Generation, Linking and also Architecture Identification activities. Properties of core Linked Data architecture depends on publishing and integration decisions on different data sources. Thus, inclusion of more than one data source is critical in terms of establishing and evaluating core architecture in early iterations. Thus, at the end of the IRS activity development team decides the stories for the iteration depending on the business and architectural perspective. 3.3 3.3.1
Architecture Identification activity is first step of the architectural
agility. Firstly, user stories of the ARC(s) that are selected in the
previous activity are analyzed to identify the architectural pattern(s)
depending on the architectural requirement of the application. From
test first perspective, test planning for architecture evaluation are
defined in this activity. Architecture evaluation is conducted by two
levels of testing. The first level focuses on the retrieving
performance of generated Linked Data and the other level focuses on
the evaluation of selected quality attibutes[
There are three well known architectural patterns for consuming
Linked Data according to application requirements: the
On-TheFly Dereferencing pattern, the Crawling Pattern and the Query
Federation Pattern [
All of them have disadvantages and advantages while making architectural decision should take into account. In On-The-Fly Dereferencing pattern, complex operations are very slow because of dereferencing thousands of URIs in the background, but stale data is never processed. The main advantage of the crawling pattern is performance. Applications can use high volume of integrated data in much higher performance than other patterns. On the other hand, the main disadvantage of this pattern is data staling and complexity of automatic linking of data on the fly. Query Federation pattern enables applications to work with current data without needing to replicate complete data sources locally. On the other hand, the main problem of this pattern is performance of the complex queries especially when query needs to join data from large number of data sources.
Also, there are a wide range of Linked Data Design Patterns
for modelling, publishing and consuming in the literature [
In the Ontology Modelling activity, concepts of domain and
relationships between these concepts are modelled and implemented
in a formal language. Ontology Modelling activity is inspired from
the following literature [
Ontology Implementation activity defines a formal ontology model from the defined conceptual model and implement it with a ontology modelling language. First sub-activity of the Ontology Implementation step is Reusability. Reusability aims to reuse known and accepted ontologies on the web. Ontology engineers try to find an ontology in semantic web search engines such as Swoogle1,
Sindice2, Watson3 and so on. If there is a ontology that overlaps outputs of conceptualizing activity, this ontology is taken as input to the following activities.
If there is not a suitable ontology, Building Concept Hierarchy
sub-activity is taken place. In this activity, ontology engineers
validate taxonomies of terms in the Domain-Property-Range Table
according to OntoClean Methodology [
Main goal of the Integration/ Modularization activity is achieving
reuse, maintainability, and evolution for large ontologies. Inspiring
from the ANEMONE[
Ontology Modelling activity focuses on the data sources that defined in the ARC(s) and generates the metadata part of the ontology for the focused data source. This activity also identifies linking requirement(s) between the ontologies in metadata level. At this point, it is possible to define test cases in the ontological level to validate consistency and competency of the developed ontologies and the linking requirement(s). The main source to define test cases is the competency questions that defined in Analysis activity. These questions are refined based on the knowledge of the developed ontologies and linking requirement(s). Also, new competency questions may be added , if they are needed. These competency questions are transfered to the real SPARQL queries to validate that developed ontology satisfies the execution of the competency questions. These queries are saved as an ontological test cases for the ARC(s) at hand. 3.3.3
Linked Data generation activity is related to generate linked data from selected data sources according to the ontology model(s). The data generation process differs according to being whether data sources structured (e.g. databases), semi-structured (e.g. XML, XLS, CVS, etc.), or un-structured (e.g. HTML, XHTML, etc.).
Structured data is mapped directly to the ontologies via RDB2RDF
convertors as D2RQ7 or Ultrawrap8. Also, “RDB2RDF Mapping
Patterns”[
ticular instructions. Except for RDF converters, Natural Language Processing (NLP) methods such as tagging can be used to acquire data from un-structured data sources.
Test case(s) to validate the ontologies are generated in the previous activity (Test Case Generation sub-activity of the Ontology Modelling). At this point, instance data is generated for the developed ontologies. Thus, it is possible to verify that real data sources are correctly transformed to the ontological instances. For this purpose, test automation script(s) that uses generated test case(s) (SPARQL queries) are written to verify that expected results is equals to the results of the SPARQL queries and these script(s) are included into Continuous Integration (CI) infrastructure. 3.3.4
One of the most important principles of Linked Data is Linking. In this activity, connections are established between data sources in manually or automatically. For this purpose, SILK12 and LIMES13 link discovery frameworks can be used in order to automatise this process. If unstructured sources like text will be linked, tools such as Dbpedia Spotlight14 can be used. Also, another method is using same URIs at ontological level to establish links manually between sources.
Similar to Linked Data Generation activity, in this activity test automation script(s) of the SPARQL queries that contains linking requirements are written and integrated into CI infrastructure. 3.4
Linked Data visualization can be a complex and time consuming task depending on the size of the visualized content. So, it is critical to start the application development cycle at the beginning of the iteration. Firstly, a visual interface draft is identified considering the ARC(s) requirements. Then, team evaluates the different layouts of the selected visualization infrastructure(s) and produces new interface design examples for the working ARC(s). These examples are discussed with the customer and then initial visual design is decided for the iteration. 3.4.2
Application does not only include user interface design components, it also includes data integration layer to handle the data coming from the Linked Data side. Real Linked Data is generated at the end of the Linking sub-activity of the Linked Data Environment Implementation cycle. But, application development should not wait till linked data generation and should continue seamlessly. Thus, development team participate into Ontology Modelling activity and/or cooperate with the ontology modelling team to develop mock linked data repository as the ontologies occures. These mock repositories give chance to work on the data integration layer of the visual design and also improve the visual components further.
In this activity, visual design and data integration layer of the design is completed by using generated mock repositories. Since, all visualization is fully functional development team can start to define accaptence test senarios. Competency questions defined in Analysis activity and refined in the Ontology Modelling activity are used to shape acceptance test senarios. Since, these senarios are defined considering the implemented visual design, developer 12 http://wifo5-03.informatik.uni-mannheim.de/bizer/silk/ 13 http://aksw.org/Projects/LIMES.html 14 https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki team implement visual test automation scripts for each senario and integrate these senarios into CI infrastructure.
In the final sub-activity of application development cycle, visual design and data integration layer of the visual design are connected with real generated Linked Data sources. Final implementation is tested with visual test scripts and whole application becomes ready for Validation&Verification activity. 3.5
Visual test automation script(s) and architecture evaluation test plan(s) are inputs of the Validation&Verification activity. All visual application test scripts are evaluated with customer and development team together to validate that these script(s) cover all ARC(s) requirements in the iteration. Then, architecture evaluation test(s) are created according to architecture evaluation test plan(s). The iteration is ended when all the defined tests are passed. 4.
In the Analysis activity, two main goals were identified by stakeholders as “Monitoring of the Container Transportation” (monitoring goal) and “Observation of the Container Transportation History” (observation goal). During the discussion, it was understood that primary customer for the system is customer operation unit employees. At this point, user stories were generated from the customer perspective. For the observation goal, stories generally focused on knowledge about the sub-transportations in the container transportation life cycle. For instance, a user story was defined as “As a customer, I want to learn history of the specific booking in a known port”. When stories were analyzed, it was seen by the development team, many stories required knowledge from different data sources. For example, to drive booking knowledge in a known port, knowledge of the agency and port data sources were needed. For the monitoring goal, stories generally define the monitoring rules like “As a customer, I want to learn specific container is loaded to a specific ship”. Thus, main senarios of these goals, their competency questions and data sources were defined as ARC(s) and added to the backlog. Figure 3 is represented an example ARC. In the IRS activity, customer indicated that observation goal is more urgent for their daily operations. Thus, it was understood that ARC(s) of these goal has higher priority. The development team outlined the criticality of the agency and port data sources, since these sources contains major part of the whole container transportation life cycle. Therefore, ARC(s) included these sources were selected for this iteration.
The development team proceeded to Architecture Identification activity and decided to use Query Federation pattern to avoid the stale data problem. Since, agency and port data sources are stored in different relational databases, the development team decided to use RDB2RDF tool for Linked Data generation as seen in Figure 3. After architecture evaluation test plan(s) were defined, federated query response time (max 0.5 second) was taken as a base criteria to evaluate retrieving performance of generated Linked Data. Also, a load test (max 1second response time for each concurrent 20 users) was planned to evaluate performance and scalability of the final application.
Ontology Modelling activity started after the Architecture
Identification. After the initial conceptualizing effort, the development
team began to search logistics ontologies that proposed in the
literature. One of these studies aims to plan military transportation
domain with different transportation types[
In the Linked Data Generation activity, data sources were
converted into RDF format by using an RDB2RDF tool according to
developed ontologies and published in a local server. These
convertions were defined in R2RML language by applying mapping
patterns in [
Since, reponse time of the federated query execution was selected as Linked Data generation evaluation criteria in Architecture Identification activity. The development team started to rework on competency questions of the selected ARC(s), revised existing ones and added new questions. Then, these questions were converted to SPARQL queries (some minor changes required in ontology model) and executed over the generated linked data. Unfortunately, architecture did not satisfy the identified performance limit in the Architecture Identification activity. The development team tried to improve query response time by using different toolsets such as Ultrawrap, D2RQ and Oracle Spatial and Graph17. Also, ontology model(s) were changed to simplfy the complex mapping(s). But, query performance was still away of expected 0.5 second and ontology structure became unrealistic from domain perspective. Thus, the development team decided to terminate the iteration and began to new one. 4.1.1
Since, learning of Linked Data development knowledge set takes time and prevent the smooth flow of activities within the iteration, plan an education for the development team about the Linked Data technologies including ontology modelling, R2RML mappings, SPARQL (as a minimum set).
Testing of Linked Data generation is conducted at the end of Linking activity. Some SPARQL queries require changes in ontologies and causes to return Ontology Modelling activity again. So, as a general rule develop test cases and execute tests when neccessary knowledge is ready. In this case, comptency questions can be revised and SPARQL queries can be generated at the end of the Ontology Modelling activity (methodology cycle was revised according to this observation). Also, seperate these test cases for single data source(s) and federation in Ontology Modelling activity and apply them in Linked Data Generation and Linking activities seperately.
Integrating architecture identification and evaluation of the identified architecture within the development life cycle is a good idea. 4.2
After experiencing the limitations of the Query Federation Pattern, the development team started to this iteration with a clear goal of changing the Linked Data architecture. Also, the development team selected a new ARC for observation goal which includes warehouse data source. This selection aimed to implement whole methodology life cycle and improve experiences and apply the learned lessons. In the Architecture Identification activity, the development team decided to use Crawling Pattern which crawls internal data sources of the company to solve the observed performance problems. At this point, solving stale data problem of the Crawling Pattern became 17 http://www.oracle.com/technetwork/database/options/spatialandgraph a problem. For this purpose, the development team decided to use a Change Data Capture(CDC)18 tool and transforming each event that comes from CDC tool to the RDF instances of the ontologies. The development team decided to choose high performance commercial CDC tool and it was planned to develop a Integration Module for transforming CDC events. Also, an open source queue implementation was selected to synchronize the CDC tool and Integration Module. It was also decided to store generated RDF in a central RDF store. General architectural view of this iteration is shown in 5. There was no change in the evaluation criteria of the architecture. Thus, architecture evaluation test plan was kept same. team decided to use AKKA infrastructure19 whis is highly scalable event driven agent system to manage concurrency, parallelism and fault tolerance. Each update actor in the Integration Module is an AKKA agent which handles XML messages in the queue and converts them to the RDF triple and updates RDF store. AKKA infrastructure of the Integration Module is represented in Figure 6.
In the Linking activity, linking was not required because of the establishing links in the ontological level. Thus, only correctness of the links between warehouse RDF data and previously generated RDF data were tested in this activity.
In the Ontology Modelling activity, the development team started to define warehouse ontology model considering previous experiences. Thus, test cases were generated at the end of this activity. The development team revised and extended competency questions of the selected ARC and tried to write SPARQL version of these questions in order to validate adequateness of the generated ontology model. After the successfully defining SPARQL queries of competency questions, the development team is started to Linked Data Generation activity. In this activity, R2RML mapping of the warehouse source was created and all converted RDF is stored to central RDF store. Then, generated warehouse data was tested by using generated test cases. Also, agency and port data were converted using previously defined mappings and stored in the central RDF store. In order to solve stale data problem, the development team started to implement an Integration Module. In this module, CDC tool handles changes that occur in the internal relational data sources simultaneous without overload. It sends each change to a java message queue (CDC Queue) using a pre-defined XML format. Integration Module is responsible for consuming XML change messages from CDC Queue. Change messages are converted to RDF triples and RDF store is updated according to found triples. However, it was realized that Integration Module should work concurently in a scalable environment, since these data sources produces approximately three million message per day. In order to handle these message traffic, the development After the successful verification of the generated Linked Data, the development team started to build a visual interface for observation of the container transportation. According to discussions with the customer, the development team identified a visual interface design and select a suitable visualization tecnique and tool. But, implementation of this interface took too long and it was not finished in the planned iteration time limits. 4.2.1
Generation of test cases and making tests in the Ontology Modelling, Linked Data Generation and Linking activities are good idea.
Implementing a visual interface after the Linked Data generation causes delay in product delivery and decreases motivation of the project stakeholders. Thus, application development should be a parallel process with Linked Data generation (Application Development cycle is added to the methodology). 4.3
In this iteration, the development team selected a new ARC for monitoring goal. Also, an additional ARC was added which includes an external data source (owned by external land transportation company) to the application, since customers wanted to observe history of the transportation life cycle with an external company. In the Architecture Identification activity, the development 18 http://en.wikipedia.org/wiki/Change_data_capture 19 http://akka.io/ team decided to implement a hybrid architecture which uses Crawling pattern for the internal sources of ARKAS and Query Federation pattern for the external land transportation company, since customer indicated that monitoring events of external company is not a high priority task from business perspective. For handling monitoring in architectural level, team thought to use another AKKA agent organization with separate queue mechanism. In this architecture, change events are transferred to new AKKA organization by the first organization and new organization agent’s creates events by applying monitoring rules. Team planned a new load test to verify monitoring with maximum daily messages and decided to observe the real systems’ CDC events to formulate the size of the load test.
After the IRS activity, the application development team started to design a visual interface for the monitoring goal in the design a visual interface for the monitoring goal in the parallel with Architecture Identification activity. The application development team worked on visual designs with the customer. In the Ontology Modelling activity, a simple core land transportation ontology was generated which is similar to the relational database schema of external company and test case(s) were generated. Also, the ontology modelling team worked with the application developers for generation of mock data in parallel. The application development team integrated this mock data with visual interface while Linked Data Generation activity was implementing. Also, visual acceptance test(s) were generated and added to the CI infrastructure. In the Linked Data Generation activity R2RML mapping of the land transportation source was defined and RDF data of the company published using an endpoint. Also, generated Linked Data was tested by applying previously generated test cases(s) and found errors were fixed. After the Linked Data generation, the application development team integrated visual application with real generated Linked Data and updated visual test(s).
In this iteration, the application was shaped around to notify user when related transportation events are occurred and represent external land transportation history in addition to previously implemented user interface. In order to catch transportation events the development team began to implement the Monitoring Module which includes the new AKKA organization. Monitoring Module responsible for catching all events in the transportation life cycle according to changed RDF triples that are explored by Integration Module. In this module, monitoring AKKA actors try to find affected transportation rules according to changes and serve them to user interface. User interface notifies users about transportation status in real time. Each change may be matched with multiple rules. Final architecture of the application is represented in Figure 7. At the end of the iteration developed application verified in Validation&Verification activity. Acceptance test(s) and load test(s) were evaluated with customer and product owner and iteration was finished successfully. 4.3.1
Integration of an external source(s) should be implemented in the early iterations of the application, since it affects architectural decisions and requires lots of effort to introduce linked data infrastructure in a new company.
Currently “Observation and Monitoring of Container Life Cycle” application is used by customer operation unit of ARKAS holding. At the end of three iterations, the developed application has generated ~300 millions RDF triples by integration agency, port and warehouse data sources of ARKAS. Also, the application handles ~3 millions change messages per day. Customer operation unit test observation and monitoring features of the application with 20 concurrent users (number of the customer operation unit employees). Observation of container life cycle features can be used by container, booking and bill of lading unique numbers. For instance, Figure 8 shows history of a booking whose number is “B14170741” in tree format (This number is correspondence of the resource whose URI is “http://data.arkas.com/general/booking/B14170741” and it is same in the RDF data of each participant company). Moreover, the development team is comfortable to work with BLOB methodology and ready to use it for other linked data project(s).
According to the feedbacks of the employees, It is realized that new ARCs are necessary to visually define new transportation rules for monitoring of the transportation life cycle. In this interface, customers can add/remove/update transportation rules that they want to follow. Also, role based access management is needed, since each customer is interested in different information about the transportation. In the future of the project, these requirements will be implemented.
The authors wish to thanks Bimar Information Technologies Services, ARKAS holding and their employees Guner Mutlu, Necmi Sentuna, Gokhan Daghan, Tugkan Tuglular, Burak Bilyay and Kubra Parkın for their helps. Also, we wish to acknowledge Galaksiya Information Technologies and Consultancy and its manager Erdem Eser Ekinci for their contributions to the developed methodology.