Figure 1: Number of credit and debit cards in circulation (2015)
Peru is recognized in the region of Latin America as a country of entrepreneurs so enterprises that have grown rapidly are more commonly found in the market; however a great number of them cease their operations in a short time due to lack of capabilities to use data to know better their clients and their competition. As a solution to help this kind of business to be sustainable in time, exists new ways to process and analyse data generated by clients through the transactions they made with their credit or debit cards. In addition, this information is usually manage by banks and financial services which with the help of new technologies as Big Data and Cloud Computing, that banks use in the daily basis, can help their corporate clients to achieve their goals by providing them with aggregated information through analytic indicators about their clients and competition.
In this article, we show the Big Data architecture of the web platform ”Crecemas”, which was developed in 16 weeks under agile methodologies with the Scrum framework and using Cloud Computing technologies. In this web, KPIs are shown using anonymized transactions about the company, its clients and its competitors, which is helpful to support commercial decisions. Currently the platform handles 200gb of information with 7 worker nodes and 3 master nodes and is used by almost 500 different companies in Peru.
The search for dynamism the generation of work
in a country is mainly to support new
entrepreneurs in order to contribute directly to
innovation
The digital transformation is a chance for the
organizations to get better at managing
information as a strategic asset and Big Data is a game
changer who adds more sources to the mix.
However, unless the lessons of the productivity paradox
are applied
Consequently, there are favorable circumstances for banks and financial services companies interested in Digital Transformation who are willing to use the information generated by the users of credit and debit card to help the smaller and newer companies to grow in the country’s economy. To accomplish this, banks can provide aggregated information obtained from sales transactions so that the business can make better decisions. 2
2.1
Hadoop is an open-source software for reliable,
scalable and distributed computing. It brings a
framework that allows the distributed processing
of large data sets across clusters of computers
using simple programming models. It is designed to
scale up from single servers to thousands of
machines, each offering local computation and
storage. The project includes these modules
This distributed framework has been adopted by
different vendors, such as Cloudera and
Hortonworks who have added important features, such as
data governance and security compliance, which
give to this powerful technology an enterprise
attractive characteristic. In addition, the community
has played an important role; for example, some of
the main tools included by the Apache Foundation
are the following:
• Apache Hive: data warehouse software that
facilitates reading, writing and managing
large datasets residing in distributed storage
using SQL
Cloud Services are applications or services offered
by means of cloud computing. Nowadays, nearly
all large software companies, such as Google,
Microsoft and Amazon, are providing this kind of
service. In addition, cloud computing has
revolutionized the standard model of service
provisioning allowing delivery over the Internet of
virtualized services that can scale up and down in terms
of processing power and storage.Cloud
computing also provides strong storage, computation, and
distributed capabilities to support Big Data
processing. In order to achieve the full potential of
Big Data, it is required to adopt both new data
analytics algorithms and new approaches to
handle the dramatic data growth. As a result, one of
the underlying advantages of deploying services
on the cloud is the economy of scale. By using the
cloud infrastructure, a service provider can offer
better, cheaper, and more reliable services. Cloud
services offer the following schemas of services
Agile methods are contemporary software
engineering approaches based on teamwork, customer
collaboration, iterative development, and
constantly changing people, process and tech. This
approach diverts from traditional methods which
are software engineering approaches based on
highly structured project plans, exhaustive
documentation, and extremely rigid processes designed
to minimize change. Agile methods are a
deevolution of management thought predating the
industrial revolution and use craft industry
principles like artisans creating made-to-order items
for individual customers. Traditional methods
represent the amalgamation of management thought
over the last century and use scientific
management principles such as efficient production of
items for mass markets. Agile methods are new
product development processes that have the
ability to bring innovative products to market quickly
and inexpensively on complex projects with
illdefined requirements. Traditional methods
resemble manufacturing processes that have the
ability to economically and efficiently produce high
quality products on projects with stable and
welldefined requirements
With all these great benefits the use of Big Data
technologies and Cloud Computing are a perfect
combination to start this journey. Also, as
mentioned in
The solution is structured in three stages that range from obtaining the data directly from the internal sources, loading them to the cloud and transforming them to be able to calculate the indicators ending in their visualization in the web (Fig.4).
The entire solution for Crecemas http:// www.crecemasbcp.com/ (Fig. 3) was developed in 16 weeks led by a scrum master and a total of 13 people dedicated exclusively where each member was grouped according to one of the three roles (Table 1).
• Business: Dedicated to engage internal areas and avoid possible business stoppers. In addition the design of the KPIs. • Data: Responsible for Big Data stage. • Development: In charge of the visualization
of the data and the web. 3.1
Data Ingestion mation from the Enterprise Data Warehouse and then store it into a file server. This processes also perform some field filtering and record according to the requirements for the KPI construction. Regarding the regulatory constraints, the main objective of this processes was to tokenize some sensitive fields that couldn’t be stored in a Cloud environment (e.g., client’s names, client’s address, card number).
In addition, each file generated has a control file to perform a validation in the data upload process.
This file contains the number of exported records and the date in which the file was processed. For this reason, data ingestion worked with two files: the first one just with credit card’s transactions data (with extension .dat) and the other one just with control data (with extension .ctrl)
The files that were extracted with these processes have the following types: • Daily master tables: That are completely
loaded. • Daily incremental tables: That have information of one day and are stored in order to have history of the data.
For this stage, different information extraction processes were built in order to obtain the infor• Monthly master tables: That are incremen
tally loaded. Business Data Development 1 Product Owner 1 Big Data Architect 1 Back-end developer 1 Navigator 2 Data Engineer 2 Front-end developer 1 Research 2 Data Expert 1 UI Expert 1 UX Expert
As can be seen in Fig. 5, The ingestion process as performed in the following way: All the orchestration of the processes in the On-Premise environment was made by the enterprise scheduling tool, which controls and executes the information extraction processes. Once this processes have finished running, one final job executes an AzCopy command (Multi-thread tool to upload data to Microsoft Azure cloud environment), which is in charge of uploading aforementioned files from the file server to the Linux servers(that were created to deploy the Big Data technology) in the cloud environment. Next, in these servers another job is executed which invokes a Ad-Hoc client that uploads the data from Linux to HDFS, to finally upload the created Apache Hive tables, which are used for the KPI’s construction. This client (Apache Hadoop and Apache Hive) was created using the Maven Artifacts from Cloudera, who is the Big Data provider selected for this project. 3.2
After the data ingestion process, data is stored in the Hadoop ecosystem (HDFS) within a Clouderas cluster which uses 7 worker nodes and 3 name nodes (1 main and 2 for backup) its architecture can be observed in Fig. 6.
This is the initial zone where the data in HDFS is located as they were loaded by the process of data ingestion. Data can be accessed through a database composed of External Tables created using HQL (Hive Query Language). First, inconsistencies in the volume of information processed by each table and business-level inconsistencies in the values are reviewed (birth date, sex, foreign characters and incongruent transactions). Then a data cleaning process occurs, which eliminates duplicates and replaces empty nulls. Finally, the data moves to a new HDFS location and is created a Hive database called Tmp Transformation Area. This area consists of two databases in Hive: the first one (Tmp Area Transformation) contains the tables generated by the previous component with potential update, adds, eliminations if necessary. The second database is the result of a transformation process of the first DB and consists of a ’Tablon’ (a large table with more than 100 columns) that consolidates all the information at the level of transactions and commerce including tokenized customer information (such as sex, age, date, educational level and economic level). This large table is used for the KPI Calculations component. In this area, the ’Tablon’ is used to generate 7 tables, each linked to one or more KPIs that are detailed in the Data Visualization part. The tables are in a Hive database, which is connected to NoSQL tables in HBase responsible to display the reports on the web.
Today we handle almost 200 gb of historical data and the entire transformation process is executed daily in 1 hour for a volume of information of approximately 12gb (11gb for master tables like customers and business and 1gb for transactions) which accumulate during the month. Also every month there is a process that goes through all the stages and takes about 1 hour with a volume of 10 gb of data (mainly other master tables related to business location, local geography).
At the steady state, daily processing should be 20gb and monthly processing 15gb which means a total of 600gb historically1 (Table 2)
12gb 20gb
10gb 15gb
200gb 600gb Data Visualization solution for this project was a web platform. (Figure 3). For that reason, this stage is a real-time request processor that allows to query the Apache HBase database, in order to obtain the data and show the results to the end user.
The workflow for this stage is shown in Fig. 7. 1Historic data from last 2 years.
This stage is composed by the following components: • External Communication: Services that allow consumption of the information in the Big Data environment. • Client Communication: In charge of establishing the remote connection with the application back-end from web pages. • Graphics: Statistics graphs on the web pages. • Web Page: Represent all the system’s web pages. • Reporting: Allows to access to the system repositories in order to perform analysis and create new reports. 4
The proposed solution allows enterprises to enable Big Data capabilities inside the organization making more efficient batch processes and reducing solutions time to market. In addition, the use of Cloud environments facilitates the adoption of technologies that require an intensive infrastructure deployment. Likewise, the agile framework used by the project, demonstrates that having the client in the center of all decisions and solutions allows a product to be created in a short time and with great value to the clients.
For future works, the process developed in
Apache Hive could be migrated to different
technologies that improves the processing speed. For
example, Apache Impala (Kornacker et al., 2015)
or Apache Spark
Apache Apache