The Ministry of Economy, Trade and Industry of Japan has launched the catalog site of their own open data at January 2013, starting with 70 data sets using customized CKAN. All of data sets are published under the Creative Commons Attribution license. After the initial publication, new data sets are added into the catalog, and the download number is also increasing smoothly. In this paper, we report the process of building this successful catalog site, and discuss the lessons learned from the project.
Linked Open Data Initiative, Inc. (LODI), which is a specific non-profit corporation in Japan, was formed in order to apply to the public call for the project fulfillment.
This first project in Japanese Government on open data was successfully finished at March 2013 in spite of the short period in a half year, and the result had an impact on not only over all Japanese Government but also the Japanese Society along with the encouraged LOD momentum in Japan.
This paper is the first report of the Open Data METI portal site7 to the Semantic Web and LOD community by who engaged in it.8 2
Up to 2012, the time of the public calling of the Open Data METI project, the effort of promoting Semantic Web and LOD had been made in long time in Japan. Especially, it is remarkable that the recent activities by the Executive Committee of LOD Challenge under the Semantic Web Conference supported by Prof. Hagino’s Lab. in Keio University has made a great contribution to LOD in Japan. The momentum is shown as the increasing number of applicants for LOD Challenge. In the LOD Challenge 2011, we had 21 works of data sets course, 34 works of idea course, and 18 works of application course. In 2012, we had 87, 50, 44, and 24 works of data sets, idea, application, and visualization course, respectively.
Furthermore, in advance to the movement in the Japanese Central Government, several local governments had tackled open data. Most of board members of LODI had deeply concerned themselves to two local governments, Yokohama and Sabae, in order to promote the LOD movement in these cities.
LODI was officially born in August 2012 as specific non-profit corporation for the purpose of spreading LOD in Japan, but the first intention was projected at a pub in Yokohama in 2010 by two key persons, a local volunteer in Yokohama, Mr. Kobayashi, and Prof. Takeda at National Institute of Informatics. So, everything had it got up to work well at the beginning of the Open Data METI project. 3
Hitachi Consulting in charge of secretariat of the IT Fusion Forum had the advantage of seeing and understanding the trend of Japanese IT and the movements of the Central Government, and then intended to acquire the Open Data METI project. They approached to LODI, who was established as specific non-profit corporation just one month before the time. Hitachi Consulting described in the proposal, “As LOD Initiative retains advanced experiences on the Open Data 7 The current web pages are shown at http://datameti.go.jp/. 8 The public announcement and a private report to the United Nations have already done by METI. Project of Sabae city, Japanization of CKAN, etc., it allows us to enable re-use of data by its competence.”
On the other hand, it was obvious for the board members of LODI that this project would be a critical point whether LOD would populate in Japan or not. Thus, the operation group of Open Data METI was formed with Hitachi Consulting, Central Research Laboratory of Hitachi, and LODI. The roles in the project of these three bodies are shown in Table 1.
Short Term Project and Different Characteristics between a Profit Organization and a Non-Profit Organization. LODI voluntarily charged themselves with the responsibility of building an open site using CKAN, because there was no software engineering body responsible for it in the group, whereas LODI had the potential on using CKAN. Fortunately, one of the board members of LODI had started the investigation of CKAN and its Japanization. He had already experiences of CKAN installation on top of his VPS servers. Then, he quickly developed the initial evaluation version on top of the personal VPS servers. Thus, it was able to show the prototype system of open data catalog site to METI persons in charge of this project at November. We used to call it the 0th version, while we called an official developing system on EC2 the Alpha version, and the system intended to run on the proper machine for Open Data METI the Beta version.
Before the notification of operation from Hitachi Consulting to LODI, we had a problem on the contract documents between Hitachi Consulting and LODI. It arose from the incompatibility of basic characteristics between a profit organization and an NPO. As a default condition on contract, Hitachi Consulting addressed the contract condition that all copyrights were reserved to Hitachi Consulting in usual custom of their own. However, it was never acceptable for LODI, who aims the promotion of LOD knowledge and technology in Japan. At the end, as Hitachi Consulting recognized the purpose of open data and this project, this problem was solved by adding a special term into the contract document.
Contents Management System. It was well known that a content management system Drupal was often used in combination with CKAN. We also regarded the appearance of sites as important. It was obvious that an attractive and flexible top page of Open Data METI was needed apart from the appearance of CKAN. We discussed in the inside of LODI about the choice of a CMS, Drupal or WordPress. Drupal by PHP is equipped with highly convenient but complex functionalities. On the other hand, WordPress is the most popular one as CMS and a member of LODI had rich experiences on WordPress. In this project, the period of engagement was very short, and it was estimated that no intimate linkage between a CMS and CKAN was required. Therefore, we decided to adopt WordPress as CMS for Open Data METI.
Water Fall Model v.s. Agile Development. In the water fall model of software development, specifications of software requirements are presented as ordering conditions by a vendee, and software vendors reply to the requirements by submitting external software specifications to the vendee, and then, both agree on the detail specifications or the internal software specifications before or after the formation of contracts. In this classical methodology of software development, a vendee is required to evaluate software specifications written on the documents. The requirements for human interface must be also made clear before the contract. Furthermore, it is usually impossible for vendors to accept drastic changes of software functionalities and interface after contract.
On the other hand, in the agile development method, frequent meetings and discussions for decision making are held in the operation between vendors and a vendee. In the case of Open Data METI, the project had to be proceeded without any precise software specification documents. Thus, we had to inevitably be in the agile development methodology. There were no software requirement specification documents and no external specification documents. So, the details of software are decided on the fly, and the software was changed step by step, seeing revised versions, starting with simple systems composed of native WordPress plus CKAN, to special systems tailored so as to meet the exposed requirements of Open Data METI.
Although many details were modified from original ones of CKAN, the most remarkable change was enabling the selection of CC licenses, including CC-BYND, for CKAN data sets and the representation of CC icons upon the web pages. 4.2
Operating System. Since ubuntu is the standard operating system of CKAN, we were anxious about the OS on Open Data METI. The Open Data METI systems were a part of larger METI web sites. At the beginning of the project, we confirmed that the OS was CentOS, and then we ported the CKAN system to CentOS 6.3, which was the latest version of CentOS at the time. However, it was CentOS 5.8 in fact. There were many differences of module compositions between CentOS 6.3 and 5.8. We solved this problem with hard work in a very short period, but this effort was useless at the end because of the memory shortage. Memory Shortage. Another big unexpected problem was allowable memory size. It is common sense for LOD researchers that the more memory the better performance. We usually use 8 GB at least and more for LOD research. However, at first it was announced to us that available memory size allocated for Open Data METI was 512MB. The reason was that they aimed the effective management of METI sites as a whole, and the software vendor who was responsible for the larger METI site did not know the common sense of LOD and no interaction between them and us. In the end, this problem for Open Data METI was hidden by the evidence of memory shortage for the whole system of METI sites, and the initial plan that the Beta version (the final version in this project) runs on the same machine as the larger METI sites was abandoned. As a result, the Beta version did not leave at EC2, which was prepared for the Alpha version (official development version in this project), and it was provided to the public on EC2 up to the end of the term of this project.9 Strong and Urgent Requirement for Beta Version from METI. The date of meeting of the Public Data Working Group strongly swayed the schedule of Open Data METI development. In the middle stage of development, METI planned to complete the functions of the Beta version for the purpose of the demonstration to the members of the Public Data Working Group. We made big efforts to meet this requirement. METI thanked us for making efforts after the demonstration. Thus, the Alpha version was finished in advance to the initial schedule plan. Requirement for Page Views and Download Numbers. At the earlier stage of development, METI required to know the number of page views and data download. Then, we enabled to know the number of access to this sight using Google Analyticator plugin for WordPress. For CKAN, we utilized CKAN Google Analytics Extension to enable the access counting. The times of pressing the down load push button is also counted. 9 The current version runs on the proper machine as a part of the whole METI site systems.
5.1
CKAN is an open software system for building catalog sites of open data, but not an RDF store for linked data. Seizing this opportune moment, LODI planned not only to build a catalog site for open data but also to realize several examples of linked data and show the potential of linked data along with providing a SPARQL endpoint. During this project in 2012, cataloged data sets were typed into two kinds of documents, white papers and statistic data. We supposed that statistic data should be appropriate to LOD. The industrial census data are published in a large number of tables. Each table shows a summary of data according to a specific view of specific aspects. These tables are categorized into 7 volumes depending on 7 standpoints.
At the first contact to experts of statistics in METI, we noticed that statistic data in the form of tables contain specific views for raw data of census. We also noticed that experts are very afraid of wrong interpretation of census data. On the other hand, we, LOD researchers, expect to show the potential of LOD by producing new knowledge in combination of distinct data in different domains. This is a sort of basic difference among professional and cultural mentalities in distinct science disciplines. It is difficult to obtain immediate consensus of opinion.
In expectation of admission from statistics experts, we made inquiries about the appropriate combinations of tables on industrial census data in 2010 that are cataloged in Open Data METI. Thus, we selected 4 tables, Industrial Fine Data by fine industrial product codes and Tokyo/Hokkaido/Osaka/Kyoto/prefectures, Industrial Semi-Fine Data by industrial semi-fine codes and Tokyo/Hokkaido/ Osaka/Kyoto/prefectures, Land and Water Data by industrial semi-fine codes and industrial districts, and Industrial Semi-Fine Data by industrial semi-fine codes and municipality. Whereby many names of properties in tables are shared, and there is no incompatibility among coarse industrial codes in 4 digits and fine codes in 6 digits of commercial products, and there is no incompatibility among municipality areas and the area components of industrial area districts. 5.2
We learned the W3C proposal of RDFization for data in table forms, RDF Data
Cube Vocabulary [
is pointed out as follows.
Generally, the advantage of RDFization of table forms
{ A datum is defined by an IRI, which provides a globally unique name.
{ A datum can unambiguously denotes a meaningful object by the help of the
strict model theory and semantics of RDF. Whereby it is enabled to process
data by machine.
{ Relations among data in distinct domains can be expressed. Whereby linked
data are available over the barrier of domains and organizations.
RDFization by RDF Data Cube Vocabulary The RDF Data Cube Vocabulary
by W3C [
{ The individual observations, and groups of observations, become (web) addressable. This allows publishers and third parties to annotate and link to this data; for example a report can refer to the specific figures it is based on allowing for fine grained provenance trace-back. { Data can be flexibly combined across datasets sets. The statistical data becomes an integral part of the broader web of linked data. { For publishers who currently only offer static files then publishing as linkeddata offers a flexible, non-proprietary, machine readable means of publication that supports an out-of-the-box web API for programmatic access. { It enables reuse of standardized tools and components.
Especially, on statistic data, the documentation of the RDF Data Cube states;
“A statistical data set comprises a collection of observations made at some points across some logical space. The collection can be characterized by a set of dimensions that define what the observation applies to (e.g. time, area, gender) along with metadata describing what has been measured (e.g. economic activity, population), how it was measured and how the observations are expressed (e.g. units, multipliers, status). We can think of the statistical data set as a multi-dimensional space, or hyper-cube, indexed by those dimensions. This space is commonly referred to as a cube for short; though the name shouldn’t be taken literally, it is not meant to imply that there are exactly three dimensions (there can be more or fewer) nor that all the dimensions are somehow similar in size.” Data Centric RDF Annotation A datum in a cell of a table can be regarded as an meaningful object with the annotations of row names at heads of rows, column names at the top of columns, titles of tables, and so on. For example, Fig. 1 shows a number of employees on food production in a industrial district area, which is annotated the row and the column names and table names. In such case, we create a cell node of RDF graph with linked annotations of a relevant area code, industrial code, and a table name, as shown in Fig.2.
The main modules of Open Data METI at March 2013 are shown in Table 2 and Figure 3. The OS was CentOS 6.3 and Apache 2.2.15 was utilized for http daemon. As mentioned earlier, we had three versions for Open Data METI after all. The 0th version was prepared on the responsibility of LODI. The machine for the Alpha version was prepared on EC2 by Hitachi Consulting. The Beta version on the proper METI machines was intended in the initial plan to be the outcomes of this project. However, the Beta version was actually implemented on EC2 and provided for public use. The actual time table is shown in Figure 4 with a number of landmark points.
The 0th version was used the reference model for discussions in operation, and also opened for the evaluation of systems at the third meeting of the Working Group. The Beta version was used for the final confirmation in the fourth meeting of the Working Group before opening to the public.
In Figure 4, the reason of gradation at “open to the public” part is the publication was gradually proceeded from the inside of the Working Group, to the limited number of Partners of Open Data METI, and an unlimited number of public people. 6.3
Unified Page Design. The unified impression of web pages is important in web design, even if two different modules are utilized. So, we created common web materials, e.g., a logo of Open Data METI, a guiding menu for navigation, which are shared by WordPress module and CKAN module. The color of web pages is also unified so as to give the same feel between WordPress and CKAN. The original page of CKAN was drastically customized. See Figure 5 as WordPress top page and Figure 6 as CKAN top page.
page for downloaded data.
Figure 7 shows an example of ranking Steering by Working Group and Create Commons Attributes License. METI organized the Public Data Working Group of IT Fusion Forum in order to discuss Fig. 5. Web Top Page of Open Data METI rules and incentives needed for promoting public open data through actually publishing METI s own data. The open license that allows reuse of data resources was deeply and earnestly discussed in the meeting of the Public Data Working Group. Two board members of LODI participated in the Working Group, and they also made a contribution to the discussion. As a result of the discussion, METI decided all data in Open Data METI should be published by CC-BY in principle except white paper documents by CC-BY-ND. Therefore, the original CKAN was customized to allow the selection of any CC licenses and enabled the CC icons including CC-BY-ND to indicate the licenses on web pages. Figure 8 shows the examples of web pages with representation of some CC license icons. 7
As summary of this report, we can conclude that there are nothing remarkable things on building an open data catalog site and a linked data site except specific knowledge and skills how to build Linked Data and CKAN. We operated the work i) with predicting problems and making decisions on the fly by agile development methodology, ii) solved occasional contingencies through efforts by relevant persons at work, and iii) met requirements from customers and the change of computational environments. It is in the ordinary way for any software developments in inexperienced fields. The success of a project is beyond one’s efforts. It was lucky for us that i) the momentum for LOD was gained in the society and ii) LODI existed at the moment. It was also happy that iii) two board
Fig. 8. Various CC icons on Open Data METI members of LODI participated in the Public Data Working Group of IT Fusion Forum. Each one of three members from LODI in the operational team made an individual contribution to the success of the project upon their own attributes. One was the best one on programming CKAN in Japan. He had experiences of CKAN installation and Japanization. Another one was a web designer who had rich experience on WordPress, and the last one who was a leader of this project in LODI once directed two national projects when he worked for a company. The construction of operational team members for the project may also be a sort of luck. Hitachi Consulting and Hitachi Central Research Laboratory also perfectly performed their part in the project.
On the other hand, we found that it is difficult for ones to understand why linked data are prominent in technologies around Webs. We often experienced that it was hard to persuade not only people in general but also ones in charge of the project without any actually working examples. For example, there is no difference in appearance between one application instance built by linked data and another application instance built by Web APIs. It is difficult to understand the potential of Linked Data from the appearance. An extreme power is required to imagine the world in which all data are linked each other and people enjoy to traverse Webs from data to data over the globe. It is also usual that a few visionaries suffers lack of understanding in the society.
The new term of Open Data METI II was already started, obtained a new member of Hitachi Systems, Ltd, who are playing a role of software development vendor. We were anxious about what went next to this project at the end of the project. We were afraid that this would become one of evaluation tests and the site would be closed. However, after the speech of METI at the end of the period of the project and METI’s private report to the United Nations, we were certain of the next stage of the project. The Open Data Charter at the G8 Summit held at Lough Erne in 2013 also pushed the Government attitude to open data.
The Open Data METI site is changed step by step at this moment. The primary mission of the new project is establishing a lasting and genuine open data site in METI. It includes new data sets, new functionalities, e.g., a bulletin board, previewing data, what’s new, and suggestion to users. Figure 9 shows the increasing number of data sets at the moment of writing this paper.
Fig. 9. Increase of Data Sets of Open Data METI
However, the most remarkable item of the development from the viewpoint of LOD is setting up terminologies of meta-tags, which are utilized in CKAN to search data sets. METI and Information-technology Promotion Agency (IPA) has started the Promotion Committee for Infrastructure for Multi-layer Interoperability (IMI) at this September, aiming the introduction of common vocabulary used in the Government beyond the boundary of Ministries and Agencies. This IMI vocabulary is going to be reflected to meta-tags in Open Data METI. In addition, a requirement of putting down in both Japanese and English expressions on the web pages is now coming up as a next requirement. LODI is again a member of the Open Data METI II operation team. We will contribute the population of LOD in Japan by making efforts to build Open Data METI and other activities.