In order to provide unified and consistent data and functionality, enterprises need to make all their applications work in an integrated manner. However, integration has become more difficult to handle when enterprises extend their business globally, more applications are implemented for supporting business activities, and more computing paradigms are adopted, e.g., cloud, social, and mobile, just to name a few. The state-of-the-art integration solutions have limitations to address the difficulty. In this paper, point to point integration engine-P2PIE, a new integration solution is presented and discussed. The solution combines distributed execution of integration with centralized management console, which innovate the fundamental integration infrastructures of point to point paradigm, and message systems. P2PIE is developed and tested by Sandvik over the decade. The solution has addressed Sandvik's challenges in enterprise application integration. Agility, cost reduction and reliability of this solution are proved.
In order to provide unified and consistent data and functionality, enterprises need to make all their applications work in an integrated manner. However, integration has been a difficult task to handle for several reasons. First, integration is highly depended on specific applications to be integrated, which involves integrating applications of various kinds, such as custom-built applications based on C++/C#, Java/J2EE, packaged applications, such as CRM or ERP applications, and legacy applications, for instances, mainframe CICS or IMS. Second, these applications are usually dispersed geographically when enterprises expend their business globally. Third, these applications are mostly operated on various platforms. Fourth, when enterprises increase cooperation and build up partnerships with other companies, there is a demand for integrating applications across enterprises.
Over the last one and a half decade, we have witnessed significant evolution in
enterprise application integration (EAI). Integration paradigms have developed from
simple file/data sharing, messaging system, message broker to more complete
integration solution, which has a centralized point of control for increasing the flexibility in
changing and managing integration [
The state-of-the-art integration solutions have limitations. This resides partly on
the integration logic for solving an integration problem that is highly distributed in its
nature by centralized software and hardware [
Sandvik, a large international company producing metal tools, headquartered in
Sweden, has managed its current and future integration challenges by developing this
new integration solution based on the business demands for agility, cost reduction,
and reliability. The solution is named Point to Point Integration Engine (P2PIE). The
P2PIE solution has been developed independently but some of its features, in
particular the point to point and message mechanisms, are similar to those state-of-the-art
solutions that suggested by [
The paper is organized as follows. In section 2, related work for enterprise application integration is briefly discussed. The case of Sandvik and its motivation for developing P2PIE are described in section 3. Section 4, P2PIE solutions are discussed, including its architecture, components, as well the gained values of implementing P2PIE. The paper ends with discussions and conclusion. Abbreviations used in this paper are given at appendix A. 2
Enterprise application integration (EAI) is aimed to make applications be able to communicate with each other. The paradigms for EAI have evolved over time, from point to point solution, to message brokers (hub-and-spoke), and to enterprise services bus (ESB)
The integration logic of a point to point integration solution is to connect two
applications which need to integrate with each other [
The integration logic of message brokers (hub-and-spoke) is built upon message
systems infrastructure [
An ESB has the same integration logic as message broker. However, ESB
transforms data and route messages by using services for achieving integration goals [
Sandvik, recognized as a world leader in tools and application solutions for the metalcutting industry, grew through many acquisitions during the period 1995-2005. A multi-brand strategy and a strong decentralization culture resulted in a legacy consisting of hundreds of business critical applications on different heterogeneous platforms and systems. These applications were communicating through various integration solutions /platforms from different vendors. Which resulted in little synergies between the main business sectors. Furthermore, most of the integration tasks were completed within each business sector by using “hard coupled” systems integration logic. Consequently, integration projects can’t be controlled within budget and completed on time. Moreover, the clear roles/mandates between central/local organizations can’t be specified, as well the central monitoring and governance of the enterprise application integration can’t be done.
Sandvik followed an approach based on Microsoft BizTalk, an ESB solution. Still, due to the increasingly higher volumes of data and the synchronous connections to and from Sandvik’s back end system, there were however still challenges to obtain a highly reliable environment. An intermediate solution was then to add an asynchronous queuing approach by IBM Webshere MQ. In this way some of the problems could be solved but also new ones were introduced, basically depending on the fact; by adding this MQ component between the sources system / receiving system and BizTalk, the control over the complete integration chain from the source system to the receiving system get lost. All these experiences called for development of a new solution that is robust, reliable, flexible, easily scalable and cost efficient.
In 2005, Sandvik decided to create a common artefact for connecting its backend systems to MQ, and this artefact would be governed and monitored centrally. While doing so, it was discovered that it would be fairly easy to make this artefact adapter based. This meant that a component was built that was distributed to all servers where it was needed, which had the capability of transferring data between any two endpoints. It also has the capability of remote management, and the source or destination endpoints could be changed from the central Management Console just by changing configurations. P2PIE solution, thus, is developed for solving EAI problems at Sandvik. 4
The P2PIE solution retains all benefits in a hub and spoke architecture by keeping management central (including configuration, control and monitoring), but decentralizing the Integration Engine in order to optimize efficiency and reduce complexity.
The key components of P2PIE solution include integration engine, central management console, and a central database. The architecture of P2PIE is shown in Figure 1.
The P2PIE is installed on the same physical servers that serves as endpoints in an integration scenario. On a conceptual level, this can be seen as a big distributed “virtual broker”, fully operated from a central management console. The individual distributed Integration Engines can act as both sources and destinations in integration scenarios.
The P2PIE in Sandvik handles the integration of applications and uses a MQ to transport messages between points. The integration line of sight is the physical reach of the integration components (integration engines). P2PIE connects business applications as endpoints directly to each other (endpoint to endpoint), thus delivering 100% line of sight. This differs from how a Broker/ESB sees the communication and integration between two applications. A Broker/ESB is put in the middle and only sees what happens with the messages up to the MQ. The line of sight is therefore limited to that point and there is no control over what happens between the application and the MQ. This is not delivering transparency into the business applications since the Broker/ESB can’t see when applications reads or puts something on the queue. This difference is illustrated in Figure 2. An “Integration Engine”, which shortly can be described as an adapter based component that is installed on all servers with software applications that need to communicate and integrate (Figure 3). The logic is thus executed in the engine and the engine itself is distributed by installing a software “integration instance”, corresponding to one service, on each server. Depending on integration scenario the server may have several instances. The source-destination connections are set up by DLL-based plugins in the integration engine. One plug-in for the source application but an arbitrary number of plug-ins for the needed destination applications. Also other types of plugins than source-destination plug-ins can be used, e.g. pre-processors or postprocessors for specific tasks. e.g. XSLT-transforming, advanced routing, or any type of enhancement to the inbound message. All plugins are stored in the central Repository and are distributed on a need-to-have basis.
Fig. 3. Distributed Integration Engine within the P2PIE solution.
Note: 1. Source, 2. Destination 1..n, 3. Pre-processor 1..n, 4. Post-processor 1..n P2PIE also provides an API so that an external application can send, or retrieve messages via a P2PIE instance. If an exception occurs, it is translated into a WMI event and can thus be monitored by SCOM. Each integration point keeps track of each event, when it starts, stops, or throws an exception and distributes the information to the central repository in nearly real time. The system monitors each step in the integration chain and alarms during deviation from the predetermined thresholds. This means that an error in the integration-chain can be quickly found and addressed to the individuals with the right skills for the task.
P2PIE is a distributed integration. However, it possesses the possibility to do service oriented integration. Since service oriented integration doesn't necessarily need to use a broker or ESB, which works well in P2PIE solution. The scalability and the resource capability of each server are not an issue in P2PIE. Sandvik monitors each server to measure its utilization. The evidence shows that capability deficiency is rarely happened in servers. 4.2
A central “Management Console”, where all integration engines are governed and monitored remotely and centrally. The GUI of the central management console of P2PIE is shown in Figure 4. The central console configure the Integration Engine, and thus integrations are created. All configurations made are stored into the central databases and to all Integration Engines that taking part in integrations. From the central management console, every execution related to an integration is controlled, including deployment, start/stop, monitoring, alerts, notifications and exception handling. 4.3
P2PIE has a central database running on SQL server. This database does not have any run-time dependencies to the distributed Integration Engines. The database saves a copy of all configurations done, a repository for all plug-ins including their history, and central log storage. • Configuration
This is where a central copy of all configurations are persisted • Repository
The repository is the storage area for all plugins that are used in the P2PIE solution. The history of plugins is also kept so that a rollback of configuration is possible. • Logs
Whenever an integration is executed, i.e. one business application sends and/or receives information from another business application, an information-log is created and stored in the central log database. P2PIE has matured and grown internally at Sandvik. It now handles all integrations where BizTalk is not involved and manages more than ten times the daily load of the BizTalk environments. This is by far the largest integration solution at Sandvik.
P2PIE is installed on more than 250 servers worldwide at Sandvik and manages more than 2,500 integration points, handling more than 500 000 transactions/day. The solution makes integration through configuration possible on the following systems and platforms: BizTalk, .NET, MS SQL Server, SAP, IBM MQSeries, Lotus Notes, Java, AS/400 and IBM mainframe.
The development and implementation of the P2PIE solution has brought significant positive effects and business value to Sandvik. The business demands for enterprise application integration on agility, reliability, and cost reduction have achieved: • Improved agility: P2PIE reduces the change and implementation times more than 30%. This is possible both as integrations are more direct between two endpoints, which leading to reduced complexity. Moreover, integration instances can be reused for configuration, which taking less time than coding again in creating an integration. • Increased reliability: P2PIE improves monitoring, enhances identification of exceptions, and handles exceptions by better and more efficient tools. The reliability of the integration related to the data flow and the content consistency have increased significantly. In 2012 the P2PIE solution handled more than 120 000 000 messages with an average of 7 exceptions per 100 000 messages. Volume evolution and reliability over time of P2PIE at Sandvik is shown in Figure 5.
Messages
Excep0ons 25 20 15 10 5 0 s e g a s s e 0m 0 00 0 /s1 n o 0 p e c x E • Reduced cost: P2PIE reduces cost in two aspects. Firstly, it reduces cost in setting up integrations in a new system development project. The integration cost in a new system development project, e.g., ERP system, was estimated at 35% of the total project budget in 2002. This was reduced to 25% 2007, and 15% 2012. Secondly, P2PIE reduces the annual running cost for operations and maintenance of the current legacy integration environment. Due to the large reduction of vendor integration platforms/suites and cost avoidance of several BizTalk instances, the percentage of the integration related cost in systems operations & maintenance was reduced from roughly 21% 2002 to 3% 2007, and even lower at 1,5% at 2012. These savings were generated from reduced maintenance costs, less hardware cost, and more reuse and configuration of implemented integrations. 5
In this paper, P2PIE, a new integration solution developed by Sandvik, is presented and discussed. The implementation of P2PIE has achieved business values for Sandvik that the company gains the capability to response changes efficiently and to pursue business opportunities effectively. P2PIE solution is aimed to lower cost, improve agility and reliability.
P2PIE integration solution is better aligned with the nature of managing a data flow across an IT landscape in enterprises. The logical flow schemes and control mechanisms are kept at a central console, while the physical point to point integration engine is implemented by taking into account the distributed nature of integration. Thereby it pursues efficient execution of the real time operational data flow. By doing so, data are allowed to flow the shortest and most cost effective way between the actual applications and databases.
Despite its simplicity and alignment with the nature of integration (with requirements for central control but distributed execution of data flow), the P2PIE solution makes significant contribution to EAI research and practice. Although the P2PIE integration solution is built upon the well- known and proven paradigm of point to point integration paradigm and message systems, the solution innovates EAI by using distributed and decentralized logic rather than the state-of-the-art centralized approach. This underlines the originality and innovativeness of P2PIE integration solution. The values achieved by implementing P2PIE at Sandvik over the years also bring strong empirical evidence for demonstrating the capability of the solution in tackling integration challenges. The novelty of the P2PIE integration paradigm is its distributed nature, where business applications send and receive data directly without passing a centralized hub, which also delivers the capability of full transparency of integration flows.
In the recent years, the need to manage a growing demand for integration is increased and intensified. Since many companies have to integrate applications used by different partners on B2B value networks, to migrating to emerging computing paradigm, e.g. cloud, mobile, social, big internet of things, cyber physical systems, just to name a few. Companies have also to meet demand for services to be available everywhere on every new devices. P2PIE is an innovative solution that can be used for addressing these integration challenges. Future research is to demonstrate and evaluate the P2PIE solution in additional instantiations both at large enterprises as well as medium sized companies.
A2A - Application to Application B2B – Business to Business CICS - Customer Information Control System (mainly run on IBM mainframe) CRM – Customer relationship management DLL - Dynamic-Link Library ESB – Enterprise Service Bus EAI – Enterprise Application Integration ERP- Enterprise Resource Planning GUI – Graphical User Interface HTTP - Hypertext Transfer Protocol IMS - IBM Information Management System M&A – Mergers and Acquisitions MQ - Message Queueing P2PIE – Point to Point Integration Engine R&D – Research and Development SaaS – Software as a service SCOM - System Center Operations Manager SOAP - Simple Object Access Protocol or Symbolic Optimal Assembly Program SQL - Structured Query Language XML - Extensible Markup Language XSLT - XML (Extensible Markup Language) Stylesheet Language for Transformations WMI - Windows Management Interface