The real-world big data are largely unstructured, interconnected, and in the form of natural language text. One of the grand challenges is to mine structures from such massive unstructured data, and transform such big data into structured networks and actionable knowledge. We propose a text mining approach that requires only distant supervision or minimal supervision but relies on massive data. We show that quality phrases can be mined from such massive text data, types can be extracted from massive text data with distant supervision, and entity-attribute-value triples can be extracted from meta-patterns discovered from such data. Finally, we propose a data-to-network-to-knowledge paradigm, that is, first turn data into relatively structured information networks, and then mine such text-rich and structurerich networks to generate useful knowledge. We show such a paradigm represents a promising direction at turning massive text data into structured networks and useful knowledge.
The success of data mining technology is largely
attributed to the efficient and effective analysis
of structured data. The construction of a
wellstructured, machine-actionable database from raw
(unstructured or loosely-structured) data sources is
often the premise of consequent applications.
Although the majority of existing data generated in
our society is unstructured, big data leads to big
opportunities to uncover structures of real-world
entities (e.g., person, company, product),
attributes (e.g., age, weight), relations (e.g.,
employee of, manufacture) from massive
text corpora. By integrating these
semanticrich structures with other inter-related structured
data (e.g., product specification, user
transaction log), one can construct a
powerful StructDB as a conceptual abstraction of the
original text corpora. The uncovered StructDBs
will facilitate browsing information and inferring
knowledge that are otherwise locked in the text
corpora. Computers can effectively perform
algorithmic analysis at a large scale over these
StructDBs and apply the new insights and
knowledge to improve human productivity in various
downstream tasks. Our phrase mining tool,
SegPhrase
The remaining of the paper is organized as follows. Section 2 introduces our recent work on automated mining of quality phrases from massive corpora. Section 3 introduces our recent studies on automated recognition and typing of entities and relations with distant supervision. Section 4 presents our initial study on meta-pattern discovery and its application to information extraction. We conclude our study in Section 5 by pointing out some future research topics on turning massive unstructured data into structured knowledge 2
Concepts are words and phrases that represent terms or ideas that people are interested in. A lot of concepts, especially scientific concepts, are in the form of phrases and are not restricted to noun phrases (e.g., “NP Complete” and “Learning to Rank”). Concepts are also often arranged in hi1http://www.yelp.com/dataset_challenge 2http://engineering.tripadvisor.com/ mining-text-review-snippets/ erarchies where each node is a topic represented by a ranked list of concepts (e.g., {‘social network analysis’, ‘mining information networks’, . . .}, is a child node of a general topic node: {‘knowledge discovery’, ‘data mining’, . . .}). Such hierarchical organization of concepts allows exploration of corpus at varied granularity, and has applications like visualization, search and summarization.
The NLP community has conducted extensive studies on automatic extraction of quality phrases, but mostly rely on many kinds of linguistic processing (e.g., chunking, dependency parsing), domain-dependent language rules, and a large amount of labeled data (e.g., treebanks).
In our recent research, we have developed
several interesting automated phrase mining methods.
The general philosophy is that instead of relying
on explicit training, we explore statistical
redundancy in document collections by frequent-pattern
mining and semi-supervised learning. Such
datadriven approaches leverage statistical or heuristic
measures derived from corpus and achieve
impressive results. Our newly developed phrase mining
approach consists of three methods: (1)
unsupervised approach (i.e., requiring neither expert
explicitly labeled training data nor knowledge-base),
represented by ToPMine
Our experiments on large text corpora show ToPMine and SegPhrase, with minor adaptation, generate quality phrases in large corpora of multiple languages (e.g., English, Arabic, Chinese and Spanish) since both methods rely mainly on statistical analysis instead of language parsing and linguistic features. For AutoPhrase, it demonstrates additional power over Segphrase on four aspects: (i) minimized human effort, using a robust positive-only distant training method which estimates the phrase quality by leveraging existing general knowledge bases; (ii) supporting multiple languages including English, Spanish, and Chinese, where the language in the input will be automatically detected, (iii) high accuracy, using a POS-guided phrasal segmentation model incorporating POS tags when POS tagger is available, and moreover, the new framework is able to extract single-word quality phrases; and (iv) high efficiency, due to a better indexing method and an almost lock-free parallelization, which lead to both running time speedup and memory saving. 3
Extracting entities and relations for types of interest from text is important for understanding massive text corpora. Traditionally, systems of entity relation extraction have been relying on humanannotated corpora for training and adopted an incremental pipeline. Such systems require additional human expertise to be ported to a new domain and are vulnerable to errors cascading down the pipeline.
Recently, we have investigated a distantly
supervised approach for extraction and typing of
entities and relations and developed several
interesting methods to reduce human effort and enhance
the performance. These include (1) ClusType
ClusType
For extraction and typing of fine-grained
entity types in conjunction with existing knowledge
bases, a major difficulty is that the type labels
obtained from knowledge bases are often noisy
(i.e., incorrect for the entity mentions’ local
context). We proposed a framework, called PLE
To Further enhance the overall performance
for entity and relation extraction and typing, We
propose a novel domain-independent framework,
called Co-Type
Mining textual patterns in news, tweets, papers,
and many other kinds of text corpora may
facilitate effective information extraction from massive
text corpora. Previous studies adopt a dependency
parsing-based pattern discovery approach.
However, the parsing results lose rich context around
entities in the patterns, and the process is costly
for a corpus of large scale. Recently, we have
proposed a typed textual pattern structure, called
meta pattern, to represent a general form of
frequent, informative, and precise subsequence
patterns in certain context. We propose an efficient
framework, called MetaPAD
Our extensive experiments demonstrate that our proposed framework discovers high-quality typed textual patterns efficiently from different genres of massive corpora and facilitates information extraction. For example, from an Associate Press and Reuter dataset (APR 2015), one can discover meta-patterns for country and president and extract country- president pairs even for rarely mentioned pairs, like Burkina Faso-Blaise Compaore´, and find which bacteria are resistant to which antibiotics from the PubMed abstracts. 5
Mining structures from massive text copora is an important task for turning big text data into big structured knowledge. Traditional approaches relying on extensive human labeling or annotation of a nontrivial sample set of documents in specific application domain are not scalable. A new direction is to develop effective weakly or distantly supervised methods to explore existing domainagnostic labels and massive existing text corpora to achieve high performance on phrase mining, entity and relation extraction and typing, and information extraction.
Our recent development of phrase mining methods, such as ToPMine, SegPhrase and AutoPhrase, entity/relation recognition and typing methods such as ClusType, PLE and CoType, as well as pattern-based discovery with massive text corpora, such as MetaPAD, contribute to this direction.
There are a lot of future research problems
along this direction. Besides further
consolidating these distantly supervised methods, an
important direction is to study automated multi-faceted
taxonomy direction from massive text to turn
extracted concepts (e.g., phrases) into organized
structures as well as identifying trusted claims and
comparative and succinct summaries, and build up
structured, multi-dimensional text-cubes and
information networks, from massive data. We have
been working along these lines and developing
some new methods, such as SetExpan
Research was sponsored in part by the U.S. Army Research Lab. under Cooperative Agreement No. W911NF-09-2-0053 (NSCTA), National Science Foundation IIS 16-18481, IIS 17-04532, and IIS17-41317, and grant 1U54GM114838 awarded by NIGMS through funds provided by the transNIH Big Data to Knowledge (BD2K) initiative (www.bd2k.nih.gov). The views and conclusions contained in this document are those of the author(s) and should not be interpreted as representing the official policies of the U.S. Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon.