In today's era of big data, increasing attention is being paid to the relationships among different types of data, and not just to those within one type of massive data, while processing and analyzing these data. To analyze and predict the trends of technologies from literatures on the basis of the conventional form of textual documents, such as academic papers or patents, the objects of analysis should include recent information collected from news websites and social media sites, which indicate the user preferences. It is necessary to systematically collect multiple texts to integrate and analyze different types of data. This study introduces practical ways to implement a database on the basis of the global standard using the unstructured information management architecture (UIMA).
At the 2012 World Economic Forum Annual Meeting in Davos, Switzerland, the big
data processing technology was highlighted as the “most important scientific
technology of the year” [
A big data processing platform consists of the four steps of collection, storage,
analysis, and visualization [
This paper explains the collection of big data related to scientific technology from different sources; this process is necessary to analyze the trends of scientific technology. It is expected to help researchers to improve their research and to better implement a database. 2
The introduction and penetration of the World Wide Web has led to an increasing
effort to crawl data from documents on the Web [
Metadata, which are needed to analyze papers related to scientific technology, were extracted from the collected data through this preprocessing procedure. The metadata were transformed into the common analysis structure (CAS) format of the unstructured information management architecture (UIMA) and were processed for implementing the data of big data texts on the basis of the global standard. 3.1
News websites are an important source for collecting the latest news on scientific technology considering the fact that there is a gap between the time of research and the publication of academic papers and patents. 154 websites, whose services include news, magazines, and forums, such as Scientific Computing1, ScienceNews2, and Bioscience3, were selected to collect the latest news on scientific technology. Data from news websites were collected using three types of crawlers, namely Google crawlers, RSS crawlers, and direct crawler, as shown in Figure 2, because they contain news published since 2001.
News articles, which were already published in 2001, were collected by using both Google crawlers and direct crawlers that collect data from the websites. In the case of a website that is run by a keyword-based search engine, the direct extraction process showed the search results when keywords related to scientific technology were used. From a website that shows lists of news articles, the crawlers extracted the links of the news. From websites that provide an RSS service, the crawlers collected links of real1 http://www.scientificcomputing.com/ 2 http://www.sciencenews.org/ 3 http://www.biosciencetechnology.com/ time news in a parallel manner. These collected URLs were stored in the URL database to avoid overlapping data. Then, the News Collector collected the HTML code of the news items through the URLs, and the News Parser extracted the title, author, date, category, and the content from the HTML code. The News Filter eliminated the overlapping and irrelevant news. 3.2
In the case of collecting foreign papers from websites such as IEEE and NCBI PubMed, data were directly collected from the websites by using the news website crawler. Meta information, which includes the title, author information, keywords, and abstract of a paper, was collected in this manner. The objects of the real-time data collection included data published between 2001 and 2014.
Patent data are also needed to analyze the levels of originality and scientific progress. Patent data released internationally, particularly in the US and Europe, and registered in the US between 2001 and 2013 were collected in bulk. The metadata and abstracts from these data were used in this study. 3.4
Wikipedia1 is an Internet encyclopedia, whose contents are created directly by the users. Here, a uniform resource identifier (URI) is assigned to every piece of information and DBpedia 2 , which provides the related meta information, is downloaded to implement the database. 3.5
Along with data from papers, patents, news websites, and Wikipedia, data from social network services were collected. Among the social media contents, we collected tweets. Tweets that included 213 keywords on scientific technology, such as web, computer, and smartphone, were collected real-time using OpenAPI released by Twitter3. Data from 2014 were collected, and on average, about 700,000 tweets were extracted daily. Punctuation marks were not eliminated in the preprocessing step, and the entire contents were stored intact as tweets in general express user emotions. 1 http://en.wikipedia.org/wiki/Main_Page 2 http://wiki.dbpedia.org/Downloads2014 3 https://about.twitter.com/what-is-twitter/
Documents collected by the Scientific Technology Text Crawling System were stored
in the CAS format of UIMA after the preprocessing procedure. UIMA is an open
Apache source project that defines the common systematic structures of software that
analyzes large volumes of unstructured information in order to discover knowledge
that is relevant to an end user. CAS is the defined form of structures that express the
feature and annotation used in UIMA. CAS is redefined and used for meeting the
characteristics of the collected metadata information. As information, which is
collected in the CAS format of UIMA, is expressed as the structure of the global
standard, it can be used as the input data for the engine that extracts unstructured
information using UIMA. The text documents collected and preprocessed in the CAS
format through this study were transmitted to the Hadoop-based information
extraction system [
In the future, we intend to study ways to apply the characteristics of the real-time data collection system, which is currently being applied only to the crawlers of Web news, foreign papers, and social media, to other types of information. Text information will also be used for analyzing and predicting the trends of scientific technology, which is necessary to help researchers to improve their research. This work utilized scientific and technical contents constructed as part of the “Establishment of the Sharing System for Electronic Information with Core Science and Technology” project (K-14-L02-C01-S02) and the “S&T Contents Construction and Service” project (K-14-L02-C01-S04). We would like to thank the Department of Overseas Information and Department of NDSL Service for providing certain contents used in this study.