Structured queries expressed in languages (such as SQL, SPARQL, or XQuery) o er a convenient and explicit way for users to express their information needs for a number of tasks. In this work, we present an approach to answer these directly over text data without storing results in a database. We speci cally look at the case of knowledge bases where queries are over entities and the relations between them. Our approach combines distributed query answering (e.g. Triple Pattern Fragments) with models built for extractive question answering. Importantly, by applying distributed querying answering we are able to simplify the model learning problem. We train models for a large portion (572) of the relations within Wikidata and achieve an average 0.70 F1 measure across all models. We describe both a method to construct the necessary training data for this task from knowledge graphs as well as a prototype implementation.
Database query languages (e.g. SQL, SPARQL, XQuery) o er a convenient and explicit way for users to express their information needs for a number of tasks including populating a dataframe for statistical analysis, selecting data for display on a website, de ning an aggregation of two datasets, or generating reports.
However, much of the information that a user might wish to access using a
structured query may not be available in a database and instead available only
in an unstructured form (e.g. text documents). To overcome this gap, the area
of information extraction (IE) speci cally investigates the creation of structured
data from unstructured content [
In the mid-2000s, with the rise of large scale web text, the notion of
combining information extraction techniques with relational database management
systems emerged [
In this paper, we present an approach that eliminates the need to have an
intermediate database in order to answer structured database queries
over text. This is essentially the same as treating the text itself as the store of
structured data. Using text as the database has a number of potential bene ts,
including being able to run structured queries over new text without the need for
a-priori extraction; removing the need to maintain two stores for the same
information (i.e. a database and a search index); eliminating synchronization issues;
and reducing the need for up-front schema modeling. [
Our approach builds upon three foundations: 1. the existence of large scale
publicly available knowledge bases (Wikidata) derived from text data (Wikipedia);
2. recent advances in end-to-end learning for extractive question answering (e.g.
[
A high-level summary of our approach is as follows. We use a publiclyavailable knowledge base to construct a parallel corpus consisting of tuples each which is made up of a structured slot lling query, the expected answer drawn from the knowledge base, and a corresponding text document in which we know the answer is contained. Using this corpus, we train neural models that learn to answer the given structured query given a text document. This is done on a per relation basis. These models are trained end-to-end with no speci c tuning for each query. These models are integrated into a system that answers queries expressed in a graph query language directly over text with no relational or graph database intermediary.
The contributions of this paper are: { an approach, including training data generation, for the task of answering structured queries over text; { models that can answer slot lling queries for over 570 relations with no relation or type speci c tuning. These models obtain on average a 0.70 F1 measure for query answering. { a prototype system that answers structured queries using triple pattern fragments over a large corpus of text (Wikipedia).
The rest of this paper is organized as follows. We begin with an overview of the approach. This is followed by a description of the training data. Subsequently, we describe the model training and discuss the experimental results. After which, we present our prototype system. We end the paper with a discussion of related and future work.
SPARQL TPF Client
QA Executor Answer Ranking
Model Runner Model Query Builder <s, p, o> to string Candidate
Selection Mo1del …
Query Service
Text Search Engine Document Col ection Col
Our overall approach consists of several components as illustrated in Figure
1. First, structured queries as expressed by SPARQL [
The candidate documents along with the lexicalized queries are provided to a model runner which issues these to models trained speci cally to bind the variable that is missing. That is given a query of the form < s; p; ?o > where s and o are bound and o is the variable, there would be speci c models trained to extract ?o from the provided document. For example, given the query (:Amsterdam :capital of ?o) we would have models that know how to answer queries of where the type of the subject is City and the property is capital of. Likewise, there would be models of that are able to answer queries of the form <?s; p; o > and so on. Each model is then asked to generate a binding of the variable. Note that the bindings generated by the models are strings. The results of each model are then ranked (Answer Ranking). Using a cut-o , the results are then translated back into identi er space and returned to the client.
A key insight of our approach is that by breaking down complex queries into triple patterns we can simplify the queries that need to be answered by the learned models.
Our approach relies on the construction of models that are able to extract
potential candidate answers from text. Following from [
Our training data is based on the combination of Wikidata and Wikipedia.
Wikidata is a publicly accessible and maintained knowledge base of encyclopedic
information [
Wikidata has a number of properties that make it useful for building a corpus to learn how to answer structured queries over text. First, and perhaps most importantly, entities have a parallel description with Wikipedia. By our count, Wikidata references 7.7 million articles in the English language Wikipedia. Thus, we have body of text which will also most likely contain answers that we retrieve from Wikidata. Second, every entity and relation in Wikidata has a human readable label in multiple languages. This enables us to build a direct connection between the database and text. Third, Wikidata is large enough to provide for adequate training data in order to build models. Finally, Wikidata provides access to their data in a number of ways including as a SPARQL endpoint, a triple patterns fragment endpoint and as a bulk RDF download. While we use Wikidata, we believe that our approach can be extended to any knowledge graph that has textual sources.
Using this input data we generate datasets of the form: [query; answer; text in which the query is answered]. As previously mentioned, complex queries can be expressed as a series of graph patterns. Thus, the queries we consider are graph patterns in which two of the variables are bound (e.g. :New England Patriots :play ?x). We term these slot lling queries as the aim is to bind one slot in the relation (i.e. the subject or the object). While we do not test graph patterns where the predicate is the variable, the same approach is also applicable. In some sense, one can think of this as generating data that can be used to build models that act as substitute indexes of a database.
Our construction method loops through all of the predicates (i.e. relations) in the dataset. It determines the frequency with which a predicate connects di erent types of entities. This is essential as large knowledge graphs can connect many di erent types using the same predicate. Thus, examples from di erent types of subjects and objects are needed to capture the semantics of that predicate. Using the most frequently occurring pairs of entity types for a predicate, the algorithm then retrieves as many example triples as possible where the subject and object of the triple are instances of the connected types - up to a given maximum threshold. Thresholding is used to help control the size of the training data.
Each triple is then used to generate a row of training data for learning how to answer graph pattern queries that contain the given predicate. To connect the graph pattern queries, which are expressed using entity IRIs to the plain text over which it should be answered, each of the components of the triple is lexicalized. The lexicalized subject and predicate of each triple are concatenated together to form a textual query and use the lexicalized object as the answer. (Note, this is trivally modi ed for the (?s, p, o case). We then retrieve the text describing the subject. We assume that the text contains some reference to the object under consideration.
The location of that reference which we term an anchor is computed by the given anchor function. For simplicity, in our implementation, we locate the rst instance of the answer in the text. This may not always represent an instance of the answer's lexical form which is located in an expression which answers the speci c question form. More complex implementations could use di erent heuristics or could return all possible anchor locations.
We apply the algorithm to the combination of Wikipedia and Wikidata dumps4. We attempted to obtain training data for all 1150 predicates in Wikidata that associate two entities together. At this time, we do not treat predicates that connect entities to literals. This is left for future work. We limited the extraction to the top 20 entity type pairs per predicate, and limited each type pair to 300 examples ). Thus, there is a maximum yield of 6000 examples per predicate. We then apply the following cleaning/validation to the retrieved examples. First, we drop examples where there is no Wikipedia page. Second, we ensure that the answer is present in the Wikipedia page text. Finally, in order to ensure adequate training data we lter out all models with less than 30 examples. Note that this means that we have di ering amounts of training data per predicate. After cleaning, we are able to obtain training data for 572 predicate for the setting in which the object is the variable/answer. We term this the SP setting. On average we have 929 examples per predicate with a maximum number of examples of 5477 and a minimum of 30 examples. The median number of examples is 312. In the setting in which the subject is the variable / answer we are trying to extract, enough data for 717 predicates is obtained. This is because the subject answer is more likely to appear in the Wikipedia page text. We term this the PO setting.
Based on the above training data, we individual train models for all predicates
using the Jack the Reader framework [
To improve performance both in terms of reducing training time and to reduce the amount of additional text the model training has to cope with, we applied a windowing scheme. This is because longer text is normally associated with greater issues when dealing with sequence models. Our scheme takes a portion of the text around the answer location chosen from the Wikipedia content. We now describe the following parameters for each architecture.
FastQA All text is embedded using pre-trained GloVe word embeddings [
JackQA Here we embed the text using pre-trained GloVe word embeddings (840 billion tokens and 300 dimensions). We use the default JackQA settings. We use a window size of 3000 characters. The batch sizes were 128/96/64 for three iterative runs. The subsequent runs with smaller batch sizes were only run if the prior iteration failed. We speci ed a maximum number of 20 epochs.
Baseline In addition to the models based on neural networks, we also implemented a baseline. The baseline consisted of nding the closest noun phrase to the property within the Wikipedia page and checking whether the answer is contained within that noun phrase.
Note, we attempted to nd functional settings that worked within our available computational constraints. For example, FastQA requires more resources than JackQA in relation to batch size , thus, we chose to use smaller embeddings and window size in order to maintain a \good" batch size.
We use 2/3 of the training data for model building and 1/3 for testing. Data is divided randomly. Training was performed using an Amazon EC2 p2.xlarge5 box. It took 23 hours for training of FastQA models, which included all models for all predicates even when there were too few training samples. For JackQA, the window was increased to 3000 characters, and multiple training sessions were required, reducing the batch size each time to complete the models which not nish from earlier runs, in all three passes were required with 128, 96 and 64 batch size respectively. Total training time was 81 hours.
Note that we train models for the setting where the subject and predicate are bound but the object is not. We also use the FastQA architecture to build models for the setting where the subject is treated as the variable to be bound. 5
Figure 3 plots individual model performance against the size of the training data given. Overall, models based on deep learning notably outperform the baseline models on average. Additionally, using these deep learning based approaches we are able to create models that answer queries for 160 additional properties over the baseline. In terms of analysis, rst, we wanted to see if there was a correlation between the amount of training data and the performance of a model. Using the data presented in Figure 3, we t a linear regression to it. We found no statistically signi cant correlation (R2 = 0:37). The model architectures show 5000 s lep4000 m a fxe3000 o r eb2000 m uN1000
0 0.0 0.2F1 Meas0u.4re (JackQ0A.6Models)0.8 1.0 0.0 0.2F1 Meas0u.r4e (FastQ0A.6Models)0.8 1.0 Models trained using Jack QA
Models trained using Fast QA. strong correlation in performance. The R2 value being 0:97 in the case of the F1 measure and 0:96 for the Exact measure. This suggests that the performance is primarily a factor of the underlying kind of data. More details are provided in Appendix A 6
To understand whether this approach is feasible in practice, we implemented a
prototype of the system outlined in Figure 1. For the triple pattern fragment
facade we modify Piccolo, an open source triple pattern fragments server to
replace its in-memory based system with functions for calling out to our QA
answering component. The facade also implements a simple lexicalization
routine. The query answering component is implemented as a Python service and
calls out to an Elasticsearch search index where documents are stored. The query
answering component also pre-loads the models and runs each model across
candidate documents retrieved by querying elastic search. We also specify a max
number of candidate documents to run the models over. Currently, we execute
each model sequentially over all candidate documents. We then chose the top
set of ranked answers given the score produced by the model. Note that we can
return multiple bindings for the same ranked results. We made some
preliminary timing estimates of a query. It takes on the order of 10 seconds to provide
results for a single triple pattern query. This is surprisingly good given the fact
that we execute models sequentially instead of in parallel. Furthermore, we
execute the models over the entirety of the Wikipedia article. Our own anecdotal
experience shows that question answering models are both faster and produce
more accurate results when supplied with smaller amounts of text. Thus, there is
signi cant room for optimizing query performance with some simple approaches
including parallelizing models, chunking text into smaller blocks, and limiting
the number of models executed to those that are speci c for the triple pattern.
Furthermore, it is straightforward to issue triple pattern fragment query requests
over multiple running instances [
Our work builds upon and connects to a number of existing bodies of
literature. The work on information extraction is closely related. [
The closest work to ours in the information extraction space is [
Like much of the work in this space our approach is based on a large scale
parallel corpus. Of particular relevance to our task are the WikiSQL and
WikiReading corpora. WikiSQL [
As mentioned in the introduction, text databases, where information extraction is combined with databases are also relevant. Our system architecture was
inspired by the pioneering work of [
Finally, there is a long history of mixing information retrieval and database
style queries together. For example, for the purposes of querying over
semistructured data [
In this work, we have explored the notion of answering database queries over text absent the need for a traditional database intermediary. We have shown that this approach is feasible in practice by combining machine comprehension based models with distributed query techniques.
There are a number of avenues for future work. In the short term, the
developed models could be expanded to include extracting properties as well as
subjects and objects. We also think that joint models for all triple pattern
predictions is worth exploring. One would also want to extend the supported queries
to consider not only relationships between entities but also to the attributes of
entities. Our current lexicalization approach is also quite simple and could be
improved by considering it as the inverse of the entity linking problem and applying
those techniques or applying summarization approaches [
In the long term, the ability to query over all types of data whether images,
structured data or text has proven useful for knowledge bases [
We looked more deeply at performance for individual models for a given property. Table 3 shows the highest performing models. We nd some consistent patterns. First, properties that have speci c value constraints within Wikidata generate good results. For example, the \crystal system" property needs to have one of 10 values (e.g cubic crystal system, quasicrystal, amorphous solid). Likewise, the \coolant" property needs to be assigned one of fourteen di erent values (e.g. water, oil, air). This is also true of \discovery method", which oddly enough is actually de ned as the the method by which an exoplanet is discovered. This is also a feature of properties whose values come from classi cation systems (e.g. \Kppen climate classi cation" and "military casualty classi cation").
A second feature that seems to generate high performing models are those that refer to common simple words. For example, the \source of energy" property takes values such as \wind" or \human energy".
Lastly, simple syntactic patterns seem to be learned well. For example, the property "birthday", which links to entities describing a month, day combination (e.g. November 8) which is thus restricted to a something that looks like a month string followed by one or two numerical characters. Likewise, the expected value for the property \ ag" often appears directly in text itself. That is the correct answer for the query \Japan ag" is \ ag of Japan", which will appear directly in text.
We also look at the lowest performing models, shown in Table 4 to see what is di cult to learn. Ratings for lms (e.g. Australian Classi cation, RTC lm rating, EIRIN lm rating) seem extremely di cult to learn. Each of these properties expect values of two or three letters (e.g. PG, R15+, M). The property \blood type" also has the same form. It seem that using character level embeddings may worked better in these cases.
The property \contains administrative territorial entity " is an interesting case as there are numerous examples. This property is used within Wikidata to express the containment relation in geography. For example, that county contains a village or a country contains a city. We conjecture that this might be di cult to learn because the sheer variety of linkages that this can express making it di cult to nd consistencies in the space. A similar issue could be present for properties such as \voice actor" and \cast member" where the values can be essentially any person entity. Similarly, \polymer of" and \species kept" both can take values that come from very large sets (e.g. all chemical compounds and all species). It might be useful for the model to be provided speci c hints about types (i.e. actors, chemicals, locations) that may allow it to nd indicative features.
Property
Fast QA Fast QA Jack QA Jack QA Training
F1 Exact F1 Exact Data Size 0.00 0.00 0.00 0.00 48 0.00 0.00 0.00 0.00 167 0.01 0.01 0.02 0.02 349 0.00 0.00 0.08 0.08 302 0.09 0.06 0.08 0.07 1838