Column type annotation is the task of annotating the columns of a relational table with the semantic type of the values contained in each column. Column type annotation is an important pre-processing step for data search and data integration in the context of data lakes. State-of-the-art column type annotation methods either rely on matching table columns to properties of a knowledge graph or fine-tune pre-trained language models such as BERT for column type annotation. In this work, we take a diferent approach and explore using ChatGPT for column type annotation. We evaluate diferent prompt designs in zero- and few-shot settings and experiment with providing task definitions and detailed instructions to the model. We further implement a two-step table annotation pipeline which first determines the class of the entities described in the table and depending on this class asks ChatGPT to annotate columns using only the relevant subset of the overall vocabulary. Using instructions as well as the two-step pipeline, ChatGPT reaches F1 scores of over 85% in zero- and one-shot setups. To reach a similar F1 score a RoBERTa model needs to be fine-tuned with 356 examples. This comparison shows that ChatGPT is able deliver competitive results for the column type annotation task given no or only a minimal amount of task-specific demonstrations.
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License conducted by Narayan et al. [17] and Jaimovitch-Lopez et Attribution 4.0 International (CC BY 4.0).
Table annotation refers to the task of discovering
semantic information about elements of a table such as columns,
relationship between columns, and entities contained in
table cells [
instance conclude that the first column should be annotated with the semantic type “RestaurantName”, while the third column containing payment methods would be labeled as “PaymentAccepted”.
the last years [
CTA methods require significant amounts of task-specific
training data. Large language models (LLMs) [
column type annotation using a subset of the
SOTAB benchmark [19].
2. We investigate the impact of providing
step-bystep instructions, using message roles, as well as
in-context learning on the performance of
ChatGPT for the CTA task.
3. We propose a two-step annotation pipeline which
enables ChatGPT to deal with large semantic type
sets.
4. We compare the performance of ChatGPT to the
performance of RoBERTa [
the experimental setup. Section 3 sets a baseline by employing three simple prompts in a zero-shot setup. In Section 4, we experiment with providing explicit instructions on how to perform CTA to the model; in Section 5, we experiment with using ChatGPT’s message roles; in Section 6, we switch to a few-shot setup and explore in-context learning for the CTA task. In Section 7, we experiment with a two-step pipeline in order to cover larger vocabularies. Section 8 compares our zero- and few-shot results for ChatGPT to the results of a RoBERTa model and DODUO. Section 9 discusses related work.
All data and code used in this paper are available at the project github2 meaning that all experiments can be replicated.
that we use for the experiments and explains how we calculate F1 based on the model’s answers.
Dataset. We use the SOTAB benchmark [19] for our experiments. The SOTAB benchmark consists of tables that have been extracted from diferent websites and are annotated using terms from the schema.org vocabulary3. The full test set of the benchmark consists of 15,040 columns and the full training set 130,471 columns which are annotated using 91 diferent semantic types. For our explorative study, we down sample SOTAB in order to keep the cost of using ChatGPT via the OpenAI API in an acceptable range4. For building our training and test sets we select tables from the original training 2https://github.com/wbsg-uni-mannheim/TabAnnGPT 3https://schema.org/ 4$0.002 per 1000 tokens. See https://openai.com/pricing and test sets. The selected tables belong to four diferent domains: music, hotels, events, and restaurants. We also down sample the label space to consist of 32 semantic types. Overall, we select 62 tables for our training set containing 356 columns which are labeled with their semantic types and 41 tables for the test set containing 250 labeled columns. We manually verify the annotations for all chosen tables. The columns contain three diferent types of values: textual, date and numerical values, with textual being the most frequent type. Table 1 provides statistics comparing the complete SOTAB benchmark datasets and our down-sampled subsets. Table 2 lists the semantic types that we use in the experiments. Note that we require the models to be able to distinguish diferent types of names, e.g. MusicRecordingName, RestaurantName, HotelName, and EventName, as well as closely related text columns such as entity description and entity review.
Language Model. We use ChatGPT version “gpt-3.5turbo-0301” for our experiments. We use the Langchain python package5 to access the model via the OpenAI API
Set
and set the temperature parameter to 0 in order to lower the variability of the answers given the same input.
Evaluation. We employ a multi-class problem setup for column type annotation, meaning that each column can be annotated with exactly one label. The metrics used for the evaluation are precision, recall and micro-F1 score. We use the micro-F1 score that is less influenced by the diferent number of examples for each label and their individual performances. The model sometimes answers using not exactly the requested terms but synonyms of the requested terms. We manually collect such synonyms from several test runs into a dictionary and count answers that are contained in this dictionary as correct in the evaluation. Altogether, the dictionary contains 27 synonyms for the 32 labels. Example of synonyms are: “Check-in Time” which can be mapped to the label “Time” or “Amenities” which can be mapped to the label “LocationFeatureSpecification”.
Designing good prompts is the key challenge for successfully using LLMs for prediction tasks as the choice of formulations [20] and even the choice of specific words [ 21] can strongly afect model performance. In order to establish baselines, we evaluate three diferent approaches to formulate simple prompts for the CTA task. The first two prompts ask ChatGPT to determine the semantic type of single columns contained in the table. The third prompt instructs ChatGPT to determine the types of all columns of the table at once. All three prompts start with a guiding sentence that instructs the model to answer according to the task given and in case that it does not know the answer, it should reply with ”I don’t know”.
Column. The column prompt uses terminology directly related to the CTA task. The prompt starts with a task description “Classify the column given to you into one of these types which are seperated by comma”. The task description is followed by the list of all the 32 types in the labels set. Afterwards, the column that should be annotated is included into the prompt. The column is represented by the keyword “Column:” followed by the concatenation of the column values in the first five rows of a table. We use the word “Type:” to instruct the model to predict the semantic type of the column. For each test example, the prediction of the model in this case is either a single word belonging to a label in the label set, a word not contained in the label set or “I don’t know”. An example of a prompt using the described format is found in the upper part of Figure 2. The example shows how the fourth column in the table in Figure 1 is passed to ChatGPT. The blue box below the prompt contains an example of an answer by ChatGPT, in this case the correct semantic type time.
Text. In order to test whether ChatGPT performs better if the CTA task is presented as a generic text classification task, the second prompt uses generic terms related to text classification. We formulate the task description part of the prompt as “Classify the text given to you into one of these classes that are separated with comma”, again followed by the list of all semantic types. The test example is again represented as the concatenation of the column values of the first five rows. In this prompt, we use the word “Class:” to instruct the model to return one of the classes. An example of a prompt using the text format is found in the middle of Figure 2.
Table. In addition to examining the content of the column to be annotated itself, it is often also necessary to consider the content of the other columns in a table in order to assign the correct semantic type to a column. In order to allow ChatGPT to exploit the context of a column for its decisions as well as allowing the model to consider dependencies between annotations, we include complete tables, inputted row by row, into the prompt and ask ChatGPT to annotate all columns of the table at once. We formulate the task as “Classify the columns of a given table with one of the following classes”. As ChatGPT version “gpt-3.5-turbo-0301” has a token limit of 4097 tokens, we select only the five first rows of a table and turn the table into string format as follows: we separate diferent cells with the notation “||” and we divide diferent rows with the notation “\n” (e.g. Table 1 would be Column 1 || Column 2 || Column 3 || Column 4 || \nFriends Pizza || 2525 || Cash Visa MasterCard || 7:30 AM ||\n...). The prediction of the model in this case returns a string separated with commas which contains in the order of the columns the type prediction for all columns in the input. In rare cases that the model replies with full sentences, the label would be contained in quotation marks, so we extract the text within the quotation marks and check if the answer can be linked to our label set using a dictionary. An example of a prompt using the table format is found in the lower part of Figure 2.
Experimental Results. The results of querying ChatGPT for all 250 columns of the test set using the diferent prompt formats can be found in the upper section of Table 3. Both text and column formats achieve a similar performance of 45-47% micro-F1 score, with text classification performing approx. 1% better. The table format scores roughly 8% less than the column format. This indicates that ChatGPT was partly confused by the longer input and the more complex task of annotating multiple columns at once.
Format column text table column+inst text+inst table+inst column+inst+roles text+inst+roles table+inst+roles
P
of the column. 4. Answer with the selected type. • text: 1. Look at the text and the classes given to you. 2. Examine the values of the text. 3. Select a class that best represents the meaning of the text. 4. Answer with the selected class. improve model performance [22]. Inspired by this work, we experiment with providing step-by-step instructions to ChatGPT on how to approach the CTA task: We ask the model to first analyze the input is it given, afterwards it should select the class/type that best represents the meaning of the input, and should then reply with the corresponding class/type. We modify our original prompt template by adding an instruction part after the task definition. For the table format, an example of an extended prompt is shown in Figure 3, while for the column and text formats we list the instructions below:
One of the important parts of these instructions in regards to the table format is that we instruct the model to generate a table out of the input that it has been given before proceeding with classification. This instruction • column: 1. Look at the column and the types was added with the motivation to make the model ungiven to you. 2. Examine the values of the column. derstand that it is working with a table and building the 3. Select a type that best represents the meaning table out of the input would give a better understanding of the rows and columns that the table is made of. The model is also instructed in the last point about the desired reply format in order to ease the parsing of the model’s answers.
Experimental Results. The results of the experiments including explicit instructions are provided in the second part of Table 3 (indicated by the “+inst” notation).
We notice that by providing instructions to the model, the performance increases by between 12 to 35% in micro-F1 score. The result that is impacted most by the instructions is the result for the table format which jumps to an F1 score of 80%. The instruction to turn the input into a table seems to help ChatGPT a lot in understanding the table content. We also observe that given the instructions, the multi-column table annotation approach clearly outperforms the two single column approaches by 18 and 22% F1.
Chat models such as gpt-3.5-turbo and gpt-4 ofer message roles to distinguish between System, User, and AI messages in a conversation. System messages are used to set the general behavior of the model; user messages are used by the user to pass a query or a task to the model, and AI messages contain the responses of the model6. The previous experiments did not use message roles. In this section we test whether using message roles improves the performance for the CTA task. As illustrated in Figure 4, we use system messages to pass task descriptions (see Section 3) and instructions (see Section 4) to the model. We use a user message to pass the actual annotation task to the model, which answers with an AI message.
Experimental Results. The results of running the experiments using the three base prompts from Section 3 together with the instructions from Section 4 and the roles described above are presented in the lower part of Table 3 (indicated with the word “roles”). From the results, we can see an increase of 28% to 39% in microF1 score compared to the column format baseline and an increase of 5% to 16% compared to the results of the instruction prompts presented in the middle section of
the table. Thus, using the message roles ofered by gpt3.5-turbo proved beneficial in all cases and we thus also use them in the following experiments.
them demonstrations of the task that they are supposed
to perform as part of the prompt [
Showing demonstrations (or training examples) to the model is also known as few-shot learning, where shots are the number of demonstrations shown. All previous experiments were zero-shot experiments as no demonstrations are shown to the model. We experiment with a one-shot and a five-shot setup. A question that arises in this case is how to choose which examples to show to the model. Since in our setup CTA is a multi-class classification problem, the training set is composed of multiple examples per label. However, we can not pick demonstrations by relevancy (e.g. show an example of restaurant name for predicting a column about restaurant names) as this would leak information about the ground truth labels. For this reason, we decide to pick demonstrations randomly from the training set, without considering the class of the entities described in the table.
In the case of column and text format, the demonstrations follow the format of the test example and therefore are columns represented by concatenating the values of the ifrst five rows of a table. For the table format an example is a randomly chosen table containing the first five rows of the original table.
As shown in Figure 5, we use a user message to present shots Table 4 prompts, on average 27 out of the 250 answers did not Average results over three runs for the three format types by exactly match the label space. Using the dictionary of providing demonstrations. “shots=5” means that five demon- synonyms (see Section 2), on average 4 of these answers stration are provided. The Δ F1 shows the diference between could be mapped to one of the labels. The amount of the micro-F1 score of our baseline model (column format zero- out-of-vocabulary answers decreases to an average of 12 shot shown in the first line) to each experiment. out of 250 in the few-shot setting, where 6 of them could be mapped to the label space.
The task description of all prompts that we presented
so far contains the complete list of the semantic types
that should be used for the annotation (32 types in our
case). Other annotation use cases might involve larger
label spaces. For instance, the complete labels space of
the SOTAB CTA benchmark [19] consists of 91 semantic
the demonstration task to the model (first user message types (see Table 1); the label space of the WikiTables
in Figure 5) and an AI message to show the model the dataset [
Experimental Results. Table 4 presents the results types that appear in tables belonging to diferent topical of running the experiments with prompts containing the domains (e.g. a hotel might have a phone number and an instructions, using message roles, and containing either address, but not a release date and an artist). The pipeline one or 5 demonstrations (1-shot or 5-shot). The reported uses two API calls: In the first step, we ask ChatGPT to scores are averages of three runs as the demonstrations predict the topical domain of the table to be annotated are randomly picked at runtime. For all three formats the (e.g. music, hotels, restaurants, or events). In the second inclusion of examples improves the performance of the step, we include only the subset of all labels which are model by 29% to 42% compared to the zero-shot column associated with the predicted domain to the task descripformat baseline. Compared to the experiments using tion and ask ChatGPT to annotate the columns of the instructions and roles (see lower part of Table 3), pro- table using only these semantic types. By breaking the viding demonstrations increases the performance by a label space in smaller spaces, we simplify the annotafurther 1-10% F1 score. Generally, we notice that with tion task as the model is presented with less and more the increase of the number of shots (demonstrations) the relevant labels to use for annotation. In the few-shot F1 score also improves with the exception of the table setup, in the first step we show tables and their domains format where we observe only a slight 0.39% increase as demonstrations, while in the second step we pick as in the 5-shot case, which might result from the model demonstrations only tables from the predicted domain. being confused by the length of the prompt including An example of the two-step pipeline is shown in Figure 5 tables. The highest increase was observed with the 6. column and text format in the 5-shot case where the performance increases by 10%. Experiments with more than ifve-shots were not conducted as the token limit of 4097 tokens was usually surpassed when showing more than 5 table demonstrations in the case of the table format.
In the zero-shot table format setting the average token length of the prompt used for annotating one test example is 550 tokens (without including the response from the model). This increases to an average of 900 in the one-shot setting and up to an average of 2320 when 5 table demonstrations are given.
Out-of-Vocabulary Answers. ChatGPT sometimes ignores the instruction to use terms from the label space but answers using diferent terms. For the zero-shot
approach are summarized in Table 5. In all cases the table classification is an easy task and achieves an F1 score higher than 95%. On average 1 error was made, and it involved a Hotel table that was predicted as an Event table. The hotel listed in this table contains the word “Park” which seems to be a word that is also used in Events.
The second step seems to achieve the highest
performance in the zero-shot setup. As in the previous table
format experiments (see Table 4) where we didn’t
notice a performance increase with the increase of
demonstrations, in this case as well we do not achieve higher
performance when showing the model 4 demonstrations.
compare the CTA results of ChatGPT to the results of
diferent baseline methods with respect to training data
eficiency. We choose three baselines that cover
diferent categories of machine learning methods: a Random
Figure 6: Example of zero-shot setup for the two-step pipeline. Forest baseline, a fine-tuned RoBERTa model [
The micro-F1 drops around 2% in the 4-shot setup. This We experiment with diferent amounts of training excould be influenced by the quality/relatedness of table amples for RoBERTa and DODUO, starting with a traindemonstrations to the test table as well as the length of ing set that contains 1 example per label (overall 32 exthe prompt. Looking in detail at F1 scores for specific la- amples) and going up to 50 training examples per label bels, ChatGPT predicts with 100% F1 the types Duration, (overall 1600 examples). The training sets with 32, 159 email, Country, currency, Coordinate, and Restaurant/- and 1600 examples are also sampled from the original name. Labels for which ChatGPT only reached F1 scores training set of the SOTAB CTA benchmark [19]. below 70% include Photograph, Rating, LocationFeature- Baseline Results. The results of the baseline experiSpecification, and Time. ments are shown in Table 6. For DODUO and RoBERTa, we report the average of the results of three runs with dif8. Comparison to Baselines ferent random seeds. The Random Forest baseline given 5 examples per label (159 examples in total) achieves a micro-F1 of 46.15% and increasing the training examples the performance of the Random Forest increases.
However, when given all the training set that contains
as BERT [
From the results, we can conclude that ChatGPT in a tions by using a cell corruption pre-training objective on
zero-shot setup without any task-specific training exam- both row and column level.
ples is capable of reaching a CTA performance that is in Data Integration and LLMs. LLMs have recently
the same range as the performance of PLM-based meth- been employed to solve diferent tasks along the data
ods given hundreds of task-specific training examples, integration pipeline. Jaimovitch-Lopez et al. [18]
exe.g. ChatGPT using the table+inst+roles prompt reaches plore the capability of LLMs to normalize diferent value
an F1 of 85.25% (see Table 3) while RoBERTa given 356 formats such as dates and units of measurement and
task-specific examples reaches an F1 of 89.73% (+4.48%). ifnd that LLMs perform comparably with systems built
The diference is shortened to 0.26% when the two-step specifically for this task. Tang et al. [ 30] and Peeters
pipeline in a zero-shot setting is used. If ChatGPT is pro- and Bizer [31] explore using GPT-3 and ChatGPT for
envided with a single demonstration the diference shrinks tity matching in zero- and few-shot setups. Narayan et
to 1.29%, e.g. the single-shot table prompt reaches an al. [17] conduct experiments with GPT-3 [
Table Annotation and CTA. Table annotation meth- fortunately, their prompt design for the schema matching
ods employ a wide range of diferent techniques ranging task isn’t included in the paper so we can’t compare it in
from statistical approaches to the more recent use of deep detail to our own prompt designs.
learning and pre-trained language models. Earlier deep
Prompt Engineering. There are many works that
learning methods like Sherlock [23] and SATO [24], use experiment with prompt engineering [
the column type annotation task. We experiment with diferent prompt designs and compare the performance of ChatGPT to the performance of PLMs-based column type annotation methods. We find ChatGPT to be much more training data eficient, requiring only a single demonstration to reach a similar performance as PMLs trained with hundreds of task-specific examples. In order to be able to deal with large label sets, we proposed decomposing the CTA task into a two-step pipeline consisting of first predicting the type of a table and second predicting column types using only the relevant subset of the full label set. As further work we plan to increase the dificulty of the test set by adding especially challenging tables from the SOTAB benchmark and to investigate how multi-step LLM pipelines as well as fine-tuned PLMs deal with these challenges.
Aji, et al., Bloom+ 1: Adding language support to Distilling BERT for natural language understanding, bloom for zero-shot prompting, arXiv:2212.09535 in: Findings of the Association for Computational [cs] (2022). Linguistics EMNLP 2020, 2020, pp. 4163–4174. [16] J. Wei, Y. Tay, R. Bommasani, C. Rafel, et al., Emer- [29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, gent abilities of large language models, Transac- et al., Attention is all you need, in: Proceedings tions on Machine Learning Research (2022). of the Advances in Neural Information Processing [17] A. Narayan, I. Chami, L. Orr, C. Ré, Can foundation Systems, volume 30, Curran Associates, Inc., 2017. models wrangle your data?, Proceedings of the [30] J. Tang, Y. Zuo, L. Cao, S. Madden, Generic enVLDB Endowment 16 (2022) 738–746. tity resolution models, in: Proceedings of the Ad[18] G. Jaimovitch-López, C. Ferri, J. Hernández-Orallo, vances in Neural Information Processing Systems F. Martínez-Plumed, et al., Can language models au- 2022 First Table Representation Workshop, 2022. tomate data wrangling?, Machine Learning (2022). [31] R. Peeters, C. Bizer, Using chatgpt for entity match[19] K. Korini, R. Peeters, C. Bizer, Sotab: The wdc ing, in: Proceedings of Advances in Databases and schema. org table annotation benchmark, in: Pro- Information Systems, 2023. ceedings of the Semantic Web Challenge on Tabu- [32] J. Zhang, B. Shin, J. D. Choi, J. C. Ho, Smat: An lar Data to Knowledge Graph Matching, 2022, pp. attention-based deep learning solution to the au14–19. tomation of schema matching, in: Proceedings of [20] Z. Zhao, E. Wallace, S. Feng, D. Klein, et al., Cali- Advances in Databases and Information Systems, brate before use: Improving few-shot performance 2021, pp. 260–274. of language models, in: Proceeding of the Interna- [33] Q. Dong, L. Li, D. Dai, C. Zheng, et al., A survey on tional Conference on Machine Learning, 2021, pp. in-context learning, arXiv:2301.00234 [cs] (2023). 12697–12706. [34] O. Honovich, U. Shaham, S. R. Bowman, O. Levy, [21] A. Webson, E. Pavlick, Do prompt-based models re- Instruction induction: From few examples to natally understand the meaning of their prompts?, in: ural language task descriptions, arXiv:2205.10782 Proceedings of the 2022 Conference of the North [cs] (2022).
American Chapter of the Association for Computa- [35] S. Hegselmann, A. Buendia, H. Lang, M. Agrawal, tional Linguistics: Human Language Technologies, et al., Tabllm: Few-shot classification of tabular 2022, pp. 2300–2344. data with large language models, in: Proceedings [22] J. Wei, X. Wang, D. Schuurmans, M. Bosma, et al., of the 26th International Conference on Artificial Chain-of-Thought Prompting Elicits Reasoning in Intelligence and Statistics, volume 206, 2023, pp. Large Language Models, in: Proceeding of the 5549–5581. 36th Conference on Neural Information Processing
Systems, 2022. [23] M. Hulsebos, K. Hu, M. Bakker, E. Zgraggen, et al.,
Sherlock: A deep learning approach to semantic data type detection, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, 2019, p. 1500–1508. [24] D. Zhang, M. Hulsebos, Y. Suhara, c. Demiralp, et al.,
Sato: Contextual semantic type detection in tables, Proceedings of the VLDB Endowment 13 (2020) 1835–1848. [25] D. Wang, P. Shiralkar, C. Lockard, B. Huang, et al.,
Tcn: Table convolutional network for web table interpretation, in: Proceedings of the Web Conference 2021, 2021, p. 4020–4032. [26] Y. Sun, H. Xin, L. Chen, Reca: Related tables enhanced column semantic type annotation framework, Proceedings of the VLDB Endowment 16 (2023) 1319–1331. [27] G. Badaro, P. Papotti, Transformers for tabular data representation: a tutorial on models and applications, Proceedings of the VLDB Endowment 15 (2022) 3746–3749. [28] X. Jiao, Y. Yin, L. Shang, X. Jiang, et al., TinyBERT: