In this paper, we present a novel and completely-unsupervised approach to unravel meanings (or senses) from linguistic constructions found in large corpora by introducing the concept of semantic vector. A semantic vector is a space-transformed vector where features represent fine-grained semantic information units, instead of values of cooccurrences within a collection of texts. More in detail, instead of seeing words as vectors of frequency values, we propose to first explode words into a multitude of tiny semantic information retrieved from existing resources like WordNet and ConceptNet, and then clustering them into frequent and diverse patterns. This way, on the one hand, we are able to model linguistic data with a larger but much more dense and informative semantic feature space. On the other hand, being the model based on basic and conceptual information, we are also able to generate new data by querying the above-mentioned semantic resources with the features contained in the extracted patterns. We experimented the idea on a dataset of 640 millions of triples subject-verb-object to automatically inducing senses for specific input verbs, demonstrating the validity and the potential of the presented approach in modeling and understanding natural language.
Most Computational Linguistics applications may need semantic information
to improve their e↵ectiveness. Semantic resources are often constructed with
automatic approaches, since manual building of ontologies is not feasible on
large scale [
Distributional Semantics (DS) is nowadays one of the frontiers in this field [
This work is based on an interdisciplinary approach that relies on Conceptual
Spaces [
An initial proof-of-concept of the proposal is given by recent research in which
the transformation of terms into top-level hypernyms carried to improvement in
several computational tasks, as in [
In Computational Linguistics, recent advances and successful experiences of statistical distributional approaches can be found in di↵erent tasks. The IBM Watson Question Answering system3 is maybe the most recent and well-known direct result. This also explains the fortunate and growing trend of available semantic resources often constructed with automatic approaches, since manual building of ontologies is not feasible on large scale. Currently, many systems actually try to automatically extract semantic knowledge from texts by means of three possible generic approaches: distributional analysis, pattern-based approaches, and Machine Learning techniques.
Nowadays, semantic information extraction is currently approached by
distributional analysis of linguistic items over specific contexts [
Distributional Semantics (DS) is nowadays one of the frontiers4 within the
Computational Linguistics field [
Current research in DS focuses on the exploration of the di↵erent impact of parameters such as context type (i.e., text regions vs. linguistic items), window (context extension), frequency weighting strategy (e.g., number of occurrences, Pointwise Mutual Information, etc.), dimensionality reduction (e.g., Latent Semantic Analysis, Random Indexing, etc.), and similarity measure (e.g., Cosine similarity, Jaccard’s coecient, etc.). Then, it produces co-occurrences matrices (or tensors) that model the semantics of the tokens by means of weights distributions.
DS refines traditional DM on text, since it considers language as a grammarbased type of data instead of simple unstructured paragraphs. However, DS still sees linguistically-refined tokens (words, lemmas, part-of-speech, etc.) as the basic bricks in VSMs, su↵ering of an intrinsic limitation: a wide range of words and grammar constructions is actually rarely used.
On the contrary, this work concerns a radical departure from this direction, releasing the assumption made by all approaches to rely on linguistic items (either terms or more context-aware tokens). The current methodology still starts
http://clic.cimec.unitn.it/composes/ by
Marco
from syntax and strings of text to extract semantics, while it would be more
reasonable to have an automated approach which also leverages on the existing
semantic resources it produces as further input. The idea at the basis of the
proposed approach is to conceptually restructure the problem of DS under the
light of research in Cognitive Science. The above-mentioned theory of
Conceptual Spaces introduced by Peter Gardenfors is about a concept representation
motivated by the notions of conceptual similarity and prototype theory [
Taking inspiration from this vision of language, as basic bricks of DS we substitute linguistic items with a representation of their meaning in terms of sets of quality dimensions. In detail, each word can be represented by a set of semantic relationships and properties that define a specific concept by means of physical and behavioral facts. For instance, a cat has legs, claws, paws, eyes, etc. (properties); then, it usually chases mouses and it sees in the dark (behaviour); it is an animal and a feline (taxonomical information), and it can have many other relations like the places where it can be habitually found.
The Conceptual Spaces (CS) framework developed in the Cognitive Sciences
field by [
In the DS current view, the triple subject-verb-object extracted from the sentence “the cat climbs a tree” is equally seen as the triple extracted from “the monkey climbs a tree”, since the two subjects share the same linguistic context. In this work, instead, the two situations will be di↵erentiated and therefore more deeply understood: in the first case, it will be able to correlate the fact of having claws with the ability of climbing a tree; in the second case, this will happen for the presence of prehensile feet. This is due to the introduction of semantics within the process of distributional analysis itself. In fact, they share physical body parts with a similar kind of functionality. Since only specific semantic information are useful at a time, this new approach can also filter out nonrelevant information (for instance, the fact that both are mammals with fur and teeth does not suggest the ability to climb a tree). Nowadays, the extraction of these features can be done due to the huge availability of semantic resources.
Once linguistic items are replaced by semantic representations, it becomes possible to reuse the methodology itself having as input the larger basis of semantic information created by the system, thus creating a positive feedback cycle and enlarging the possibilities of the system. We call this concept as semantic loop, and, to the best of our knowledge, it is the first attempt to go beyond single-processing systems that connect syntax with semantics towards recursive processing of extracted semantics. For example, the action of “seeing” can show a correlation with the fact of having eyes. Nowadays, the link between actions and properties of subject and objects are not used while they actually provide significant information for deeper language understanding.
This paper presents an approach that also relies on the concept of diversity.
Diversity has been taken into account mostly in Information Retrieval (IR)
scenarios, where systems become aware of the need of obtaining search results that
cover di↵erent aspects of the data [
This shift in the basic bricks opens new research questions and challenges concerning Data Mining methodologies: the problem of correlating atomic linguistic items becomes to correlate sets of features, where only some of them are actually significant. Thus, the new challenges become to understand: – which features need to be filtered out – which features can be combined to approximate concepts (according to Conceptual Spaces) The advantages of the proposed research direction are the following: – the integration of semantic information within the internal steps of the current methodology can create a virtuous loop through which semantic resources can be automatically extended. – linguistic items are fragmented into minimal conceptual information that enables statistical approaches to overcome the problem of low-frequency words occurrences. In fact, even in very large corpora, there is little chance of finding statistically-significant patterns that can be used to carve out meanings out of text. This is known as the long tail problem. Statistical approaches are usually not able to propagate existing information belonging to frequent data to such long tail. One of the aim of this proposal is to define a linguistic framework in which both rare and frequent words are fragmented into more basic facts on which reason on, avoiding low-frequency issues. – the use of multilingual resources will have an impact on the creation of more powerful semantic resources, that will be more independent by word-to-word translations. Within the DS field, a minimal step in this direction has already been done by means of transformations of words into general concepts or super-senses. However, this only involves terminological abstractions by means of IS-A relationship substitutions. In fact, our proposal represents a generalization of this since it considers a wider spectrum of conceptual relationships. For example, a person can assume the role of living entity, doctor or student in the context of breathing, making surgical interventions, and studying mathematics respectively. The point is that only specific properties are activated by the context at a time, so we avoid to assign fixed top-level hypernyms for all the cases. In addition to this, the simple generalization of a linguistic item does not extend the current analysis of correlations between atomic tokens.
The outcomes of such novel approach can be many: – a new methodology that introduces the concept of semantic loop, i.e., iterative use of extracted semantics as input for further extension of semantic resources – new semantic resources, created by the use of the proposed methodology – revisitations of Data Mining techniques for dealing with a new and more complex type of data with respect to standard VSMs applied on text – the proposed contribution can also have impact on how semantic knowledge can be re-used or inherited from data in di↵erent languages. For instance, in case there is no translation for two words in two di↵erent languages, it will be possible to leverage their semantic information to link them automatically. Only translation at concept level it will be needed (i.e., translation of the new feature space). Thus, the semantic loop can work also for alignment of di↵erent languages. 3
Our proposal concerns an automatic methods to build a large-scale semantic framework based on a concept-level distributional analysis of the semantics contained in plain texts. Our methodology avoids manual constructions of ontologies which is known to be unfeasible. On the contrary, the method goes towards a direct and extensive exploitation of the wealth of available resources regarding semantics. In particular, it leverages di↵erent types of knowledge that can be used to transform words (intended as lemmas or generic linguistic items, from now on) into sets of extended and fine-grained semantic information. The resulting explosion of such heterogeneous knowledge, coming from di↵erent sources and methods, create a new challenge: how to align, filter, and merge it in order to feed Vector Space models with semantics, as opposite to lexical entities. 3.1
In this paper, we started focusing on ConceptNet [
In spite of this, the approach can work with other resources. For example,
another type of knowledge that can have an high impact on our semantic
integration comes from Linked Open Data (LOD). One of the most used LOD resources
in Computational Linguistics is DBPedia, a dataset containing data directly
extracted from Wikipedia. It contains more than 3 million concepts described by 1
billion triples, including descriptions in several languages. Other knowledge bases
are UMBEL (i.e., a 20k subjects ontology derived from OpenCyc), GeoNames
(i.e., descriptions of geographical features), and several others. Then, WordNet
[
In order to experiment the validity of the approach, we had the need of computing a distributional model starting from a large collection of texts. However, instead of parsing corpora from scratch, we used a dataset of subject-verb-object (SVO) triples generated as part of the NELL project5. This dataset contains a set of 604 million triples extracted from the entire dependency-parsed corpus ClueWeb09 (about 230 billion tokens)6. The dataset also provides the frequency of each triple in the parsed corpus. We integrated a Named Entity Recognition module to transform proper names into generic semantic classes, like people and organizations7. 3.3
In this section, we explain the details of the approach. In particular, the
algorithm is composed by three di↵erent phases: (1) the data pre-processing step
with the generation of two transactional databases (transactions of items, as in
the fields of Frequent Itemset Mining and Association Rules [
the generation of the semantic vectors, i.e., vectors whose features represent conceptual and semantic facts rather than document- or context-occurrences. Since the aim of the system is to capture senses from data, we start from the root of the meaning, that is the verb. So, for a specific input verb v, we parse all the SVO triples in the datasets that have a frequency higher than a set threshold8, and we only take those who are morphological variations of v. Then, for each one of these triples, we query ConceptNet with the subject-term and
whole ClueWeb09 corpus, in order to remove noisy data. the object-term, retrieving all their semantic information that will later build the new semantic space. Table 2 shows an example of the information collected in this phase.
Then, we associate each semantic information to a unique id and construct two transactional databases: one for the semantic information of the subjects, and one for the objects. An example of result of the first phase is shown in Table 3.
Diverse Itemsets Mining Once the transactional databases are built for a specific verb “v ”, we use techniques belonging to the field of Frequent Itemset Mining to extract frequent patterns, i.e, semantic features that frequently cooccur in our transactional databases.
The description of the problem is the following: let I = i1, i2, ..., in be a set of items (i.e., our semantic features) and D be a multiset of transactions, where each transaction t is a set of items such that t ✓ I. For any X ✓ I, we say that a transaction t contains X if X ✓ t. The set X is called itemset. The set of all X ✓ I (the powerset of I) naturally forms an itemset lattice. The count of an itemset X is the number of transactions in D that contain X. The support of an itemset X is the proportion of transactions in D that contain X.
An itemset X is called frequent if its support is greater than or equal to some given percentage threshold s, where s is called minimum support.
When the database contains a significant number of large frequent itemsets,
mining all of them can be very expensive, since the space of itemsets to generate
can be huge. However, if any subset of a frequent itemset is frequent, it can be
sucient to discover only all the maximal frequent itemsets (MFIs). A frequent
itemset X is called maximal if there does not exist a frequent itemset Y such that
X ✓ Y . Mining frequent itemsets can thus be reduced to mining a “border” in
the itemset lattice. All itemsets above the border are infrequent and those that
are below the border are all frequent. Another type of frequent itemset, called
closed frequent itemset (CFI), was proposed in [
In our experimentation, we used the library called SPMF for finding closed
frequent itemsets9, applying the CHARM algorithm [
In order to produce these novel types of frequent itemsets, we viewed the
problem as a set cover problem [
subject-itemsets with all the object-itemsets, weighting the connection according to the their co-occurrences in the same triples of the original dataset.
The semantic verb model constructed for a specific verb “v ” is thus a set of weighted connections between frequent and diverse semantic features belonging to the subjects of “v ” and frequent and diverse semantic features of the objects of “v ”. On the one hand, this is a way to summarize the semantics suggested by the verb. On the other hand, it is also a result that can be used to generate new data by querying existing semantic resources with such semantic subjectand object-itemsets. Still, this can be done without looking for words similar to frequent subjects and objects, but by finding new subjects and objects that, even if not similar in general, have certain semantic information that fill a specific context.
The resulting models are automatically calculated, and they are very concise,
since in all the large and sparse semantic space only few features are relevant
to certain meanings (headed by the verb). This is also in line with what stated
in [
In this section we present the result of the approach on di↵erent cases. In particular, we extracted all the triples in the dataset containing di↵erent verbs like to play, to eat, to sing, and so forth. Then, for each of these verbs we executed the algorithm and extracted the models, i.e., sets of weighted pairs of diverse subject- and object-itemsets. Table 4 shows some examples of the automatically extracted semantic information.
In the experiments, we wanted to evaluate the quality of the constructed models and their ability to generalize over the input data also taking into account their size in comparison with classic word-based vector spaces.
On the one hand, the approach is able to model the meanings expressed by complete verbal phrases with minimal resource requirements, as shown in Figure 1. In fact, starting from hundreds of verbal instances, the method produces itemsets with a feature space much smaller then common word spaces in which words and chunks are represented by vector spaces of the order of thousands of features. For instance, in the presented example, with a minimum support of 0.05 (i.e., 5%), the resulting model is constituted by 4 diverse itemsets for the objects and 24 for the subjects, with an average itemset cardinality of 18.5 and 12.6 respectively, covering more than 50% of the semantic features of all the input triples.
On the other hand, we calculated the coverage of the extracted models, that is the percentage of triples subject-verb-object in the input data in which at least one item is included in the extracted diverse itemsets. These results are shown in Figure 2. Notice that the coverage of the diverse itemsets is always equals to the coverage of the closed itemsets, even if the formers are less than (or equal to) the latters.
To the best of our knowledge, this is the first attempt to model entire linguistic constructions subject-verb-object in terms of senses at sentence-level automatically carved out from the data by deeply analyzing co-occurrent fine-grained semantic information instead of lexical and syntactic chunks. We think that further e↵orts on this direction can importantly change the vision and the horizon of current Natural Language Understanding goals as well as the management of large collections of textual data with concise, generative, and semantics-based models. 5
This contribution represents a first e↵ort to pass from standard word-vectors to semantic vectors. This causes the raise of new challenges, like the alignment and the filtering of heterogeneous semantic information. Still, such shift in the basic bricks also concerns Data Mining techniques, since the problem of correlating linguistic items becomes to correlate sets of semantic features, where only some of them are actually significant. In this paper, we presented an approach that connect Natural Language Processing techniques (Lexico-syntactic analysis, syntactic parsing10 and Named Entity Recognition) with Pattern Mining approaches like Frequent Itemset Mining and the cover set problem.
To produce semantic vectors, we started by using ConceptNet, one of the largest semantic resource currently available. In spite of this, in future work we will also come back to lexico-syntactic parsing of large corpora like Wikipedia for the extraction of further semantic information directly from text.
The impact of this new research direction can be extremely high. The main question this proposal wants to engender is the following: what if computational systems can directly reason on semantics instead of syntax? Future NLP 10 We refer here to the used subject-verb-object input structures. technologies could move away from language through more complex meaning understanding, also dealing with unseen and low-frequency words.
By reducing commonly-huge vector spaces based on linguistic items into synthetic conceptual matrices, we also attack the Big Data problem for textual databases. For example, if we think at the term “color ”, a linguistic-based vectorial representation would contain hundreds of terms that usually co-occur with it, such as “pastel ”, “dark ”, “light ”, “red ”, “brilliant ”, and so forth. In Wikipedia, for instance, we found more than 500 adjectival lemmas that co-occur with this term. On the other hand, the concept of “color” can be potentially represented by few dimensions. For instance, the HSV scheme uses only three dimensions: brightness, hue, and saturation.
We evaluated the approach by its ability to reduce the space and generalize over the input data. In future work, we will also measure the approach on tasks like Ontology Learning and Question Answering. This paper also introduces a the concept of semantic loop, i.e., the recursive use of extracted semantics as input for further extensions. The use of this methodology can create new and extended semantic resources.
Finally, we will leverage techniques for data compression like Multi
Dimensional Scaling (MDS) [