Information and communication technologies (ICT) are changing research methods in social sciences, especially in the ways of getting data. Online surveys and Big Data analyses are used the most among them. Using Open Data is another way realized by the spread of ICT. Using Open Data by administrative agencies has potential but also some difficulties. This study discusses the practical use of Open Data, and focuses on problems related to data in Portable Document Format (PDF) files. Those problems seem to occur because many officers in charge of Open Data do not pay attention to the principle and practical use of Open Data. It shows a gap between drafters and practitioners in our society.
If questionnaire was made well with options or Likert scales, it is possible to collect numeric data, as the results of such surveys are categorical and quantifiable. In the past social sciences have often relied on physical or offline questionnaire surveys.
Nowadays, online surveys with questionnaire are also a popular way for gathering user data. Respondents can answer to the questionnaire with smart phones or tablet terminators. However, there have been problems in online surveys compared with offline surveys. Typical cases are summarized in Table 1. The online survey respondents are automatically limited to the Internet users who have registered to a survey company. We cannot reach those who do not use the Internet and those who are not registered to any Internet services. In addition, online direct mails are easy to be ignored. It leads to low recovery rate. Therefore, online surveys have been regarded being biased in sampling.
On the other hand, conventional offline surveys also got problems recently.
Mail survey in Table 1 has been an efficient data collection tool since 1788 [
When almost all the people come to use the Internet, online survey may have
less sampling bias than conventional survey. Visiting respondents in offline
survey can secure reliable response, while online submission can reduce Hawthorne
effect that respondents tend to give desirable answers, which let them look more
normative [
With improvement of ICT and computing, we came to treat Big Data, that is,
huge amount of transactions of information. Including GAFA (Google, Amazon,
Facebook, Apple), many large corporations utilize Big Data which they collect in
their businesses [
Companies are eager to protect their algorithms, but at the same time, the
data itself can contain sensitive information, for instance transaction data or
personal customer information. That is why many countries are developing legal
systems on personal information (Table 2). Japan amended the Act on the
Protection of Personal Information for use of Big Data. It includes “Clear Indication of
the Purpose of Use”, “Consent of the Person on Provision to A Third Party” and
“Anonymization of Information”. General Data Protection Regulation (GDPR)
in EU has more rules. It is said that DPA in Kenya and PDP in India are based on
GDPR [
USA Japan China EU Kenya India *amended
In June 2013, Hitachi announced that it would start a service that utilizes the
boarding / alighting history of JR East (Japan’s largest railway company)’s Suica
(IC prepaid fare card) as big data and provides it as station area marketing
information. At first, JR East claimed that it was not disclosing personal information
on its customers, but admitted selling data without their consent and apologized
after a month [
In January 2021, messaging app WhatsApp announced the new Privacy
Policy, which will allow WhatsApp to share data with its parent company, Facebook.
It does not apply in EU, because it violates GDPR [
In March 2021, LINE, which is very similar to WhatsApp and dominant in
Japan, let Chinese engineers at a Shanghai affiliate access Japanese users’ data
without informing them [
Big Data can be international. It is important to pay attention to the latest
trend in the world. Even if the use of data is legitimate in Japan, it may violate
GDPR in EU [
We have already covered some of the challenges when it comes to
collecting numeric and large amount of data in social sciences. Using Open Data could
present an alternative way of data collection. Knowledge is open if anyone is free
to access, use, modify, and share it [
Using academic Open Data is, in other words, the secondary use of data. The data of GSS (General Social Survey) in USA are generally available in formats designed for statistical programs, and “GSS Data Explorer” allows users to test hypotheses, and look for interesting correlations directly on the website.
Social data obtained by administrative agencies are also published and free
to access by the public. According to a questionnaire survey in Japan, medians
of Open Data government possession rates were only 1% to 5% in each section:
spatial Information, Agroforestry, Commerce and Industry, Medical and Welfare,
Education Tourism, and Others [
By way of illustration for a problem in Open Data by administrative agency,
there have been arguments on official announcement about the results of national
academic ability survey in Japan. When the governor of Osaka prefecture,
Hashimoto decided to publish the data by cities, towns and villages, some
municipalities and activists were against it. When he became the mayor of Osaka city, he
disclosed the results of the city by schools. Now results in 2011 and 2012 are
available except municipalities with only one school [
As mentioned at Introduction, Open Data by administrative agencies are not only numeric but can also come in the form of documents. They are provided in PDF (portable document format) files. There were 9776 PDF data sets in Japanese data catalogue site “DATA.GO.JP”. They enabled cross-sectional search of the data by the central government [14]. That made up about 40% of all the data sets in the site. After 4 years, while the data sets increased by 50%, the rate of PDF format keeps still 41.2% of all (Fig. 1).
These PDF files often are not machine-readable, even if they have literal or numeric data. When we retrieve them, we may need OCR (optical character recognition) software. For example, Election Commission of India has data of donation, which have tables with donors and amounts. However, they are not machine-readable. After retrieving data with software, we have to review the error rate of the OCR algorithm, with viewing operation. They are scanned data from paper documents, which were printed out. Punch holes in sequential documents often damage some parts of data. The spread of paper-less transactions in administrative agencies may solve these problems.
All the above plus, Use open standards from W3C (RDF and SPARQL) to identify things, so that people can point at your stuff
Berners-Lee, known as the inventor of the world wide web, developed star rating system “in order to encourage people -- especially government data owners -- along the road to good linked data” [15] (Table 3). This table present a scale, well known to officers in charge of Open Data in governments.
PDF format is supposed to be worth 3 stars in a manner independent of
application software, hardware, and operating systems. However, many data in PDF
will not get even 2 stars because they are not machine-readable. In the survey on
local governments in Japan [
Though Berners-Lee drew a blueprint of Linked Open Data, the star scheme he developed does not cover all current problems connected to open data. For instance, to get Linked Open Data, we need more Open Data. The value of data will increase, if there are more related data [16]. However, those data are not easy to use, unless they are machine-readable. If they are not easy to use, people may not use them. There is a structured interview research, which showed data users could motivate officers in charge of Open Data [17]. Emotion of officers cannot be overlooked. The above-mentioned officers also said, “It is pleasure for public servants that people use Open Data. It encourages us to contribute for public interest.” They know how many times their Open Data sets were downloaded. If people use more data, officers may publish more data (Fig. 2). The difference between 1 star and 2 stars is very important as well as that between 0 star and 1 star. Here are proposals to Star scheme.
• There should be an instruction to distinguish non-machine-readable PDF from machine-readable PDF. • Machine-readable should be translated to “possible to copy and paste textual data or matrix data” for ordinary people. • PDF should be included as examples, as well as CSV and excel. • “2 stars system” can highlight the importance of the difference between 1 star and 2 star. It can be more efficient to encourage officers in charge toward Linked Open Data, so far. • Open Data providers can share 3 to 5 stars works to the third parties or
Open Data catalogue site. 3
This study highlights some of the many challenges involved in collecting numeric data in social sciences, the actual conditions of Open Data by administrative agencies, and a practical use of Open Data. There are changes in research methods with ICT. Conducting a social survey is getting difficult. Utilizing Big Data for academic purpose still presents many problems, which need to be solved. At the same time, using Open Data by administrative agencies has a tremendous potential. Open data is a source for a high volume of free documents. Many document data are made by scanning printed documents. They are published in non-machine-readable PDF files. People may not pick up such Open Data, which are hard to use. More use of Open Data can generate more Open Data from public sectors. Therefore, some proposals on the problem of Open Data in PDF files were presented. Though the problem of a file format in this study looks very trivial, it may have prevented the spread of Open Data. The Star scheme was made to encourage officers in charge of Open Data. It has been well known to them, its principle still does not seem to be realized by them even after a decade. It must be significant to have pointed out a gap between drafters and practitioners in our society.
This study only pointed out the existence of the problem of PDF. It was discussed only with cases in Japan and India. It was not examined whether the problem exists all over the world, and how many non-machine-readable PDF there are. There can be some reasons that officers in charge tend to make image PDF files. For example, they may be going to put priority on signatures or stamps. Convenience is not always right. It should be discussed with Electronic Signature together. These points should be improved and will be the future works. 14. Honda M.: The whole aspect of public data to suppose from “DATA.GO.JP”. Journal of Japan
Society of Information and Knowledge, 26(4), 320–325, (2017) 15. Berners-Lee, T.: Linked Data. (2006). https://www.w3.org/DesignIssues/LinkedData, last accessed 2021/03/30 16. Shapiro C.: Information rules : a strategic guide to the network economy. Varian, Hal R. Boston,
Mass. Harvard Business School Press (1999) 17. Honda M., Kajikawa Y.: Importance of communication between policy makers and external actors in the policy formation process. Proceedings of the 15th National convention of Japanese Association for Communication, Information and Society. pp. 204-207 (2018)