High-quality Web tables are rich sources of information that can be used to populate Knowledge Graphs (KG). The focus of this paper is an evaluation of methods for table-to-class annotation, which is a sub-task of Table Interpretation (TI). We provide a formal definition for table classification as a machine learning task. We propose an experimental setup and we evaluate 5 fundamentally diferent approaches to find the best method for generating vector table representations. Our findings indicate that although transfer learning methods achieve high F1 score on the table classification task, dedicated table encoding models are a promising direction as they appear to capture richer semantics.
Tabular data is one of the most prevalent data representations. The efort by Cafarella [
Due to the inherent scarcity of labelled data for the first sub-task (class-annotated tables), a table classification model must either be of low complexity (few parameters) or leverage pre-trained models. Using pre-trained models in TI has been studied only to a very limited extend. Hence, we explore two promising directions for making learning-based approaches more eficient: (a) by using transfer learning, (b) by considering additional inductive biases that are unique to tabular data representations.
We propose an experimental setup with the intention of finding the best method for
generating a representation which captures the information from the table but also the row and column
structure, so that it can be later used towards solving the remaining sub-tasks of TI: row-to-entity
linking, column type annotation and relation extraction. We are interested in understanding
how pre-trained language models, such as BERT [
• An extensive empirical evaluation of the diferent approaches.
In this section, we review prior work related to solving the diferent sub-tasks of TI. We also
give a short overview of methods for generating vector representations of tables.
Table Interpretation The three sub-tasks of TI were first introduced in the paper by Ritze
et al. [
To the best of our knowledge, the T2K Matcher is the only existing method for solving the table-to-class task. Namely, the class of the table is chosen by ranking the sum of the similarity scores of the column-to-property correspondences aggregated per class. Since this method requires querying of the KG for candidate retrieval and first solving the column-to-property alignment in order to find the correct class of a table, we do not consider it during our experiments. In contrast to the T2K Matcher, we consider a closed book scenario, where the instances of the KG are not available, only the classes in the KG schema. Representation Learning on Tables
Based on powerful LM, dedicated deep learning models
have recently been proposed to exploit tabular data structures, e.g., in table-based question
answering [
We focus on the task of table-to-class annotation. The task has been introduced together with
the two other TI sub-tasks in [
An entity table is a × matrix where and are the number of rows and columns of the table . Each element of the matrix , , , contains one or more tokens, where each token
1,∗, 2,∗, … , ,∗ . is a sequence of characters. We denote with ,∗ and ∗, the -th row and the -th column of the matrix respectively. The header of the table is the first row = 0,∗. The content of the
Let = {( 1, ), … , ( , )} be the set of labeled tables with number of tables, and each label ∈ is in the set of classes defined in the KG schema = { is a model, with a parameter vector , which encodes each table 1, … , }. A table encoder ∶ { } → ℝ to a vector ( ) = and = { 0, 1, … , } is the set of feature vectors for every ∈ . The final task is to train a classification model
∶ ℝ → so that each table vector is assigned to one of the class labels. The problem is defined in the multi-class setting. Formally our setting is ∘ ∶ { } → , where only the parameters are trained on the table classification task, i.e., no gradient updates are performed on .
textual corpora and an approach for question-answering which has been pre-trained on tabular data (Figure 1 (b)). The code for the experiments is accessible online 1.
For evaluation we used the second version of the T2D gold standard dataset [
In the evaluation we used 5 diferent models as table encoders, varying from general purpose document encoders to more sophisticated LM, pre-trained on tabular data.
TF-IDF
or term frequency-inverse document frequency, is a term weighting scheme which generates vector representation for a document based on the frequency of the words in the document. It is the simplest method which we used as a table encoder.
Spacy pre-trained word vectors on a text extracted from blogs, news and comments. We used
the vectorizer from english-medium sized pipeline3 which contains vocabulary of size 684830.
Word2Vec pre-trained word vectors trained with FastText 4 on a Wikipedia text corpus. The
model used for the learning the vectors [
BERT is a widely used, Transformer-based LM [
4.3. Setup To systematically evaluate the quality of the representations generated with the diferent table encoders, we compare their performance on the classification task under diferent scenarios. It is important to note that we did not train or fine-tune any of the methods for table encoding, i.e., we used them of-the-shelf . Since the tables can be large, in order to avoid scalability issues, we resort to sampling of rows. Namely, we first shufle the rows in the tables and then we sample the first rows. The shufling of the rows is done only once. For the experiments, we sampled ∈ {1, 3, 5, 7} rows from each of the tables and used these sampled tables as input to the table encoders.
When using TF-IDF as table encoder, the input is a set of sequences, where each sequence feature vectors tf-idf ∶ → . corresponds to a table from the set of tables . More formally, a table sequence for table is a sequence of rows = ( 0,∗, 1,∗, … , ,∗ ), such that ∈ {1, 3, 5, 7} , and the set of sequences is the set = { 0, … , }. The table encoder TF-IDF transforms the set of table sequences to the set of table content are concatenated into one vector = ‖ .
Word2Vec and Spacy generate the vector representation for table in 3 steps. First, the sequence , representing the header of the table , is encoded as the mean over the word vectors in the sequence , represented as . Second, the content of the table, is transformed into a table sequence = ( 1,∗ … ,∗ ) and encoded as the vector , which represents the mean over all the word vectors in . Finally, the vector representations for the header and for the
Considering that there is a limit on the length of the sequence that BERT can encode in one step, we used diferent transformation for the last two methods. BERT encodes each table row by row, i.e, a sequence ,∗ is generated for each of the rows ,∗ of table , where 0 ≤ ≤ . BERT generates row-wise vectors, so for each sequence ,∗ the output is a vector ,∗ . The vector representation for table is the vector which is the result of the mean-pooling over the set of the BERT’s output vectors { 0,∗, … , ,∗ } that correspond to the table rows. In the same manner, the TaBERT model also first generates an encoding for each of the rows of table resulting in a set of vectors. This model uses vertical self-attention focused on the vertically stacked vectors, { 0,∗, … , ,∗ }. Because of the vertically aligned vectors, the output of the model wiseadcoolmu meann-vpeoctoolirnrgeporveesretnhteatcioonlu{m n∗,0r,e…pr, e s∗e,n t}aftoiornesactoh goefntehreate tchoelutambnles einnctoadbilne g . F.inally,
We then use the Multi-layer Perceptron (MLP) with one hidden layer of size 500, the tanh activation function and adam optimizer as the classifier from Figure 1. The hyper parameters are chosen after an extensive search and they are fixed for all of the experiments. Since the available dataset is small, instead of splitting it once into a training set and a test set, we use stratified K-fold validation with = 20 splits. Considering that the dataset is imbalanced, we report the macro averaged F1 score. The reported scores are the average of the results on the test set after the cross validation. To explore the efect of the column names, we also encoded the tables with their column names masked. Specifically, for all of the tables, we substitute their column names with the token [UNK].
TaBERT Analysis To get a better understanding of the (under-) performance of TaBERT we analyse the influence of the utterance and its interplay with column names. In addition to the empty string “ ” used in previous experiments, we also used a randomly generated string with 10 characters (unique per table), and one constant string, Thing, for all tables. Moreover, we experimented with adding the correct class of the tables as utterance, as well as a wrong class (for instance, all the tables of class Country are encoded with the class Plant as utterance). Figure 3 shows the results of these experiments, where the input tables were with = 3 rows. The horizontal axis shows the diferent options that we passed as utterance to the model and the vertical axis shows the achieved F1 score. The masking of column names has significant influence on the generated table representation. The reason for this might be in the way how a row is transformed into a string, i.e., the value of each table entry is concatenated with the column name of the entry and its value. Observing the results with the diferent utterance, we see that the choice of utterance does not afect the performance of the model when the column names are not masked. Nevertheless, when the column names are masked, the influence of the utterance is more significant. In both cases when the utterance is the wrong class or the correct class, the achieved score is much higher, which might be attributed to a class-wide shift in the vector space because of the grouping that these utterances cause.
In this paper we explored diferent types of table encoders for generating vector representations for tabular data. Specifically, we focused on evaluating diferent methods for table encoding on the sub-task for TI, table-to-class annotation. Despite the increasing interest in the problem of TI, so far, only one approach towards this specific sub-task has been proposed. In this direction, we provided a formal definition for the table-to-class annotation task as a machine learning task. We conduct an empirical study with five diferent methods for generating vector representation of a table and evaluate their performance on the table-to-class annotation task. The results from our experiments show that transfer learning methods with large vocabularies of pre-trained word embeddings perform on par with more complex and expensive modes such as LM pre-trained on tables. An interesting finding is that the inductive bias for tabular structure in TaBERT did not bring benefit to the performance of the BERT model. A possible explanation for this is the missing significant utterance that the TaBERT model expects as input. Nonetheless, the miss-classifications made by this model are reasonable, suggesting that the vector representations capture the semantics of the tables. Future work should target closing the gap between existing general-purpose models and model specific for encoding tabular data. To further our work we plan to explore other existing methods for table encoding for solving the table-to-class task, as well as for solving the entity-to-row and column-to-property tasks.