=Paper=
{{Paper
|id=Vol-3180/paper-39
|storemode=property
|title=Fraunhofer SIT at CheckThat! 2022: Semi-Supervised Ensemble Classification for Detecting
Check-Worthy Tweets
|pdfUrl=https://ceur-ws.org/Vol-3180/paper-39.pdf
|volume=Vol-3180
|authors=Raphael Antonius Frick,Inna Vogel,Isabella Nunes Grieser
|dblpUrl=https://dblp.org/rec/conf/clef/FrickVG22
}}
==Fraunhofer SIT at CheckThat! 2022: Semi-Supervised Ensemble Classification for Detecting
Check-Worthy Tweets==
https://ceur-ws.org/Vol-3180/paper-39.pdf
Fraunhofer SIT at CheckThat! 2022: Semi-Supervised
Ensemble Classification for Detecting Check-Worthy
Tweets
Raphael Antonius Frick1 , Inna Vogel1 and Isabella Nunes Grieser1
1
Fraunhofer Institute for Secure Information Technology SIT | ATHENE - National Research Center for Applied
Cybersecurity
Rheinstrasse 75, Darmstadt, 64295, Germany
https:// www.sit.fraunhofer.de/
Abstract
During the corona pandemic misinformation has been increasingly spread on social media. Since the
automatic verification of social media postings has shown to be challenging, there exists the need of
classification systems, that can identify check-worthy posts in social media feeds. In this paper, the
classification system used in the CLEF2022-CheckThat! Lab to detect check-worthiness in English tweets
is presented. The proposed system, that took advantage of ensemble and semi-supervised learning
showed promising results in the experimental evaluation. Further, first experiments were conducted
with the novel framework lambeq to solve the classification problem using quantum natural language
processing (QNLP), which were not part of the final model. The final model ranked fifth best in terms of
the F1-score in the competition.
Keywords
check-worthiness detection, Twitter, semi-supervised learning, GAN, QNLP
1. Introduction
Within several weeks after the outbreak of the coronavirus in 2019, fake news about
the topic started to spread on the internet. The term fake news hereby refers to the
targeted distribution of misleading or false information to influence a persons’ opinion
or behavior. Especially social media platforms and instant messengers have played an
extensive role for the exchange and distribution of coronavirus related fake news. The
information shared by humans manually or automatically through social bots on social
media covered several topics, such as conspiracy theories about the origin of the outbreak
and the existence of the virus, the negative effects of vaccines, as well as advertisements
for untested or even harmful treatments. Since some of the misinformation can have an
impact on a persons’ life, there exists the need to detect whether a tweet or other social
media postings contains false statements or claims.
However, even with the current advancements in natural language processing, the
CLEF 2022: Conference and Labs of the Evaluation Forum, September 5–8, 2022, Bologna, Italy
Envelope-Open raphael.frick@sit.fraunhofer.de (R. A. Frick); inna.vogel@sit.fraunhofer.de (I. Vogel);
isabella.grieser@sit.fraunhofer.de (I. N. Grieser)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution
4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
automatic verification of social media postings remains an open issue, as they can not
be classified with high accuracies [1]. The main objective of the subtask 1A on the
CLEF-CheckThat! Lab [2, 3, 4] is to automatically determine the check-worthiness of
tweets. The results of such an analysis could be used to create a set of tweets, that need
to be further investigated by a human to verify their claims.
In this paper, a novel approach to detecting check-worthiness in tweets, that was
developed in conjunction with the CheckThat! Lab, is presented. It combines multiple
state-of-the-art transformer networks to an ensemble classification scheme and takes
advantages of semi-supervised learning assisted by Generative Adversarial Networks
(GAN). Furthermore, first experiments using Quantum Natural Language Processing
(QNLP) and the framework lambeq were conducted. The proposed classification system
ranked fifth place in the English 1A subtask.
The paper remainder is structured as follows: at first, an introduction to related work
presented in the previous iteration of this competition is given in Section 2. Then, in
Section 3 the task and the associated datasets are presented. In Section 4 an overview of
the proposed approach is given along with explanations on the applied data augmentation
and preprocessing. Section 5 showcases and discusses the experimental results achieved
on the test set. The paper then concludes in Section 6 with an outlook for future work.
2. Related Work
The detection of check-worthiness in tweets has been a substantial part of the CheckThat!
Lab over the past years [5, 4]. The lab held in 2021 already dealt with detecting the
check-worthiness of tweets in the context of the pandemic. Back then, the best performing
group proposed the usage of an ensemble classifier [6] consisting of BERT, ALBERT,
RoBERTa, DistilBERT and Funnel-Transformer fine-tuned on a corpus featuring check-
worthy and non-check-worthy tweets. The team that came in second [7] took advantage
of heavy data augmentation, such as machine-translation and word substitutions. Their
model was based on a fine-tuned Bertweet on the augmented data. The team that
ranked third in the competition analyzed the semantic similarity estimation provided by
a fine-tuned sentence transformer [8].
3. Task & Dataset Description
This year’s CheckThat! Lab task 1 [3] revolved around identifying relevant claims in
tweets. The task itself was divided into several subtasks, with the objective of task 1A
being to predict, whether a given tweet is worth fact-checking. While the task covered
several languages, such as Arabic, Bulgarian, Dutch, English, Spanish and Turkish, the
authors only participated in the English variant of the subtask.
Table 1
Class distribution of the CheckThat! Lab 2022 task 1A English dataset and of the CheckThat! Lab 2021
task 1A English dataset
2022 2021
ℒ0 ℒ1 ℒ0 ℒ1
Train Split 1675 447 532 290
Validation Split 151 44 80 60
Validation-Test Split 445 129 331 19
Sum 2271 620 2271 620
3.1. 2022 Dataset
Along with the lab, a dataset was released. It consisted of tweets mentioning coronavirus
centered topics collected in a time frame between 2020 and 2022, which were then
manually labelled by humans. Tweets that were mapped with a label of ℒ0 represented
tweets that were not check-worthy, while those labeled as ℒ1 were considered check-
worthy by the annotators. The dataset was divided into four data splits, a train split, a
validation split, a validation-test split and a test split. For all splits, except the test split,
the annotations were released ahead of the lab evaluation phase. The label distribution
showcased in Table 1 suggest that the dataset suffers from heavy class imbalance. Thus,
this needs to be considered during training.
3.2. 2021 Dataset
Additionally, to the data from the current CheckThat! Lab, the dataset of the previous
iteration of the lab [4] served as training data for the semi-supervised GANs and were
used to train the meta classifier of the ensemble classifier. Some samples can be found in
the dataset from 2022 and 2021. Thus, they were removed from the 2021 splits to avoid
any duplicates. Similar to the 2022 dataset, this dataset was also heavily imbalanced
regarding the class distribution (Table 1).
4. Methodology
In this section, the methods serving as candidates for the detection pipeline are presented.
The overall concept used for the final submission is visualized in Figure 1. It consists
of an ensemble classification scheme that takes advantage of data augmentation during
training as well as several preprocessing steps.
4.1. Data Augmentation
As observed in Section 3, the training data suffers from data scarcity and class imbalance.
To tackle the issue of data scarcity, the training data underwent a data augmentation
process before applying any preprocessing.
Figure 1: Visualization of the check-worthiness detection pipeline used in the final submission
Bert-Base-Cased
Bertweet
Check-
Worthy
Data- Twitter-Roberta-Base-2021
Tweet Back-Translation SVM
Augmentation
Not
Data-Augmentation
Check-
(training only)
Worthy
Bert-Base-Cased
(GAN)
Twitter-Roberta-Base-2021
(GAN)
Ensemble Classifier
Back-translation was applied to generate new training samples. In comparison to
synonym exchange, back-translation has the advantage that it introduces newly crafted
sentences that syntactically differ a lot from the original sentence, but stay semantically
very close to them. Thus, back-translation for the languages Spanish and Chinese was
considered by first translating the tweet from English to the target language and then
back to English using Google Translate. Chinese was considered for making the model
more robust to samples that contain language errors. An example is showcased in Table
2. The utilization of back-translation resulted in an enriched dataset featuring three
times the size of the original dataset.
Table 2
Result of back-translation applied on a sample tweet
Original Morning Headlines: Canada is marking 1 year of the global pandemic,
the latest on vaccines in Ontario, and Metrolinx gets a payment update.
https://t.co/EzHH8iArkl
Spanish Titulares de la mañana: Canadá cumple 1 año de la pandemia mundial,
lo último sobre vacunas en Ontario, y Metrolinx recibe una actualización
de pago. https://t.co/EzHH8iArkl
Back-Translated Morning headlines: Canada marks 1 year of global pandemic,
vaccine update in Ontario, and Metrolinx gets paid update.
https://t.co/EzHH8iArk
4.2. Preprocessing
Before feeding the tweets into the classifiers, they need to undergo several preprocessing
steps. In comparison to news articles or speeches, sentences of social media postings often
do not follow correct grammar and moreover often feature an extensive usage of slang
words. Furthermore, emojis are used to further express emotions and special control
characters, such as @ and #, are used to identify user mentions or hashtags on Twitter.
For this purpose, the Python package pysentimiento[9] was utilized. It contains
functions that allow exchanging emojis with their corresponding descriptive tokens. It
also allows identifying URLs, user mentions as well as hashtags. Since URLs do not
contain any information that drives the check-worthiness of a tweet, they were removed.
User mentions were generalized by exchanging them with a @user-token. The result on a
sample is displayed in Table 3.
To solve the issue of the imbalanced class distribution of the dataset, random under-
sampling was utilized. This resulted in the removal of randomly selected samples assigned
to the majority class until the overall class distribution in the datasets were balanced
out.
Table 3
Result of the preprocessing applied on a sample tweet
Original: Over 75% of #Covid19 vaccinations carried out in just 10 countries
Vaccine monopolies are creating artificial scarcity.
We urge members of the @wto to support the #TRIPSwaiver
proposal and help boost global vaccine production.
Our statement https://t.co/JYr7MMbSzR https://t.co/Cm5G8k4yJ7
Prepocessed: emoji warning emoji Over 75% of covid19 vaccinations carried out
in just 10 countries ! Vaccine monopolies are creating artificial scarcity.
We urge members of the @USER to support the tripswaiver
proposal and help boost global vaccine production.
Our statement emoji right arrow emoji emoji left arrow emoji
4.3. Ensemble Classifier
The ensemble classifier consisted of five weak models, that were fine-tuned to solve the
check-worthiness detection task. A meta-classifier is then used to conduct weighted
decision-making. Three of the weak classifiers were trained in a supervised fashion, while
two of them took additionally advantage of semi-supervised learning. The models that
were chosen were determined based on the achieved F1-scores on the test-validation split.
For supervised training, the following base-models were used:
• Bert-Base-Cased [10]: The BERT model was pretrained on a corpus consisting of
11,038 unpublished books and English Wikipedia articles. For the classifier, the
case-sensitive model was considered, as sometimes emotions can convey through
uppercase writing.
• Bertweet [11]: Bertweet is a language model based on RoBERTa, that was trained
on a large-scale Twitter corpus featuring 845M tweets collected within a timeframe
ranging from 2012 to 2019 as well as 5M tweets related to the COVID-19 pandemic.
Thus, this model already has an extensive vocabulary of coronavirus related tokens.
• Twitter-Roberta-Base-2021 [12] Similar to Bertweet, Twitter-Roberta-Base is model
based on RoBERTa that was trained on a twitter corpus. The corpus hereby
consisted of 123.86M tweets gathered in 2021.
Semi-supervised learning was realized with GAN-Bert [13]. GAN-Bert is a generative
adversarial network, that synthesizes a set of artifical vector embeddings, which allows
fine-tuned models to achieve high accuracies when trained on scarce labeled data. It
also enables the utilization of unlabeled samples in the training process. Hereby, the
aforementioned Bert-Base-Cased [10] and Twitter-Roberta-Base-2021 [12] models served
as base-models.
Each of the weak classifiers were trained on the train and validation split of the 2022
dataset using 10-fold cross-validation. This allows the models to be trained on as much
data available as possible to identify optimal parameters as part of grid search. The
models were trained for 15 epochs. To avoid overfitting, the use of model checkpoints
discarding any model except the best performing one on the validation split has been
considered. The learning rate was initially set to 0.00004, as lower learning rate resulted
in bad convergence, while larger values worsened the validation scores. However, by using
Adam as the optimizer, the learning rate is constantly adjusted during training.
During semi-supervised learning, the validation-test split of the 2022 dataset as well
as the train, validation and test splits of the 2021 dataset were fed into the classifier
as unlabeled data. Similar to the supervised learning classifiers, the GAN-Bert models
were also trained using a 10-fold cross-validation. This results in a total of 5 × 10 = 50
trained weak classifier. A meta classifier, here an SVM, takes the probabilities retrieved
from the weak classifiers and outputs a combined classification decision. In comparison
to averaging the probabilities, this has the advantage, that the single probabilities get
weighted based on the classifiers individual performances.
4.4. Quantum Natural Language Processing Modelling (QNLP)
A new method to deal with classification problems in the natural language processing
field is quantum natural language processing (QNLP). One library that supports QNLP
is lambeq [14].
Figure 2: Example of a string diagram annotated with pregroup grammar
In lambeq, sentences are represented as string diagrams (Figure 2), which can be
further formulated as tensor networks. These describe the relationships between the
different words in a sentence, which are defined according to the pregroup grammar
formalism. Lambeq supports the generation of two types of models: quantum-based
models and classical models, which involves training classical tensor networks. For this
competition, a classical model was trained to analyze the efficiency and usefulness of the
framework.
There are three steps when creating a model in lambeq. First, the sentences need
to be transformed into their corresponding sentence diagrams. This is done by using a
parser, which calculates the syntax tree of the sentence and converts it to a sentence
diagram. Then, a circuit is built based on the sentence diagrams. The form of the circuit
differs depending on whether a quantum model or a classical model is built. At last, a
model can be built based on the circuits of all sentence diagrams. This model can then
be trained.
5. Experiments
Table 4 displays the classifications scores achieved by each model on the test set. The
system used for the final submission consisted of an ensemble classification system
featuring a meta classifier taking advantage of five weakly fine-tuned base-models. The
QNLP-based model implemented using the framework lambeq posed several issues and
thus was not included in the final submission.
Table 4
Results on the test set. Values displayed in bold indicate, which fine-tuned base-model was able to
achieve the best score for a particular metric
Accuracy Precision Recall F1-Score
min max min max min max min max
Bert-Base-Cased 0.6980 0.7517 0.3778 0.5263 0.0256 0.5128 0.0487 0.5194
Bertweet 0.6242 0.7651 0.3559 0.600 0.1538 0.6154 0.2449 0.5333
Twitter-Roberta- 0.3557 0.6308 0.2857 0.3947 0.5641 0.9744 0.4418 0.5246
Base-2021
Bert-Base-Cased 0.4228 0.6711 0.3089 0.4375 0.7949 0.9744 0.4691 0.5882
(GAN)
Twitter-Roberta- 0.4026 0.6107 0.3015 0.3913 0.6923 1.000 0.4606 0.5496
Base-2021 (GAN)
Lambeq - 0.7210 - 0.3570 - 0.3020 - 0.3270
Probability Averag- – 0.6174 – 0.3929 – 0.8462 – 0.5366
ing
Meta Classifier – 0.6846 – 0.4394 – 0.7436 – 0.5524
5.1. Supervised Models
The models, that were trained in a supervised manner, consisted of a Bert-Base-Cased,
Bertweet and Twitter-Roberta-Base-2021 model, by fine-tuning them on the train and
validation split of the 2022 dataset using 10-fold cross-validation. The weak classifiers of
the Bert-Base-Cased model were able to achieve the highest minimal accuracy across all
considered models. However, it did not perform well in terms of the achieved F1-score in
comparison to the other models. The Bertweet model that was fine-tuned in supervision,
maintained the highest accuracy and precision scores. It was also the best performing
supervised classifier concerning the F1-score. The trained Twitter-Roberta-Base-2021
was able to achieve the highest recall score among the three. However, it did not perform
well in terms of accuracy, precision and F1-score.
5.2. Semi-Supervised Models
In addition to the models that were trained in a supervised manner, experiments with
GAN-Bert were conducted, which fine-tuned a Bert-Base-Cased and a Twitter-Roberta-
Base-2021 model on labelled and unlabeled data. While accuracies achieved by both
semi-supervised models were lower than their supervised counterparts, these models
excelled in achieving by far the highest recall and F1-scores. Thus, this shows the
effectiveness of semi-supervised learning for solving classification problems in natural
language processing and especially in the detection of check-worthiness of tweets.
5.3. Ensemble Classification
For retrieving the final classification decision from the 50 weak classifiers, two types of
ensemble classification were considered. The simplest approach is to average the proba-
bilities gained from classifier and then estimation the class with the highest probability.
This has the advantage that it does not need any additional data and does not increase
the inference time. A major disadvantage, however, is that one bad performing weak
classifier may have a massive impact on the overall classification result. Thus, a linear
SVM was used, serving as a meta-classifier. While this requires additional data to train
the meta-classifier, it conducts a weighted decision-making. This has the advantage that
it is more robust to single classifiers that did not perform well. This advantage also is
visible in the classification results on the test set, where the meta-classifier is able to
achieve higher accuracy, precision and F1-scores than when taking solely advantage of
probability averaging.
5.4. QNLP Model
For this competition, the usage of the new library lambeq posed some practical difficulties.
First, the framework is not designed to handle multi-sentence inputs. This means that
for tweets with multiple sentences, the data needs to be divided in its separate sentences.
Furthermore, the default parser in lambeq, the BobcatParser, can not parse the tweets
reliably in a structure which is necessary for the model creation and model training
process. This makes the preprocessing and model creation process more difficult.
The results of the lambeq model were in total worse than the results of the final
model. The results can be seen in Table 4. Thus, it was not included as part of the final
classification system.
6. Conclusion
In this paper, a novel approach to detecting check-worthiness in tweets has been proposed.
It takes advantage of state-of-the-art transformer models, such as BERT and Bertweet
and combines supervised and semi-supervised learning by leveraging from ensemble
learning. Moreover, there have been made first experiments on the utilization of QNLP
to solve this classification problem. The evaluation had shown that QNLP and the
framework lambeq still pose many issues. This might be connected to the fact, that both,
the technique and the framework, are still fairly new at the time of writing. However,
experimenting with semi-supervised learning, such as GAN-Bert, revealed promising
results, as those models were able to achieve greater F1-scores than their counterparts
learned in a supervised manner. Furthermore, taking advantage of a meta-classifier
instead of averaging probabilities has shown to yield better results, as bad classifiers will
not have a considerable influence on the final decision. Future work could revolve around
taking advantage of other semi-supervised training techniques, such as virtual adversarial
training.
Acknowledgements
This work was supported by the German Federal Ministry of Education and Research
(BMBF) and the and the Hessian Ministry of Higher Education, Research, Science and the
Arts within their joint support of “ATHENE – Disinformation on Corona” and “Lernlabor
Cybersicherheit”.
References
[1] G. K. Shahi, J. M. StruSS, T. Mandl, Overview of the CLEF-2021 CheckThat! lab
task 3 on fake news detection, in: Working Notes of CLEF 2021—Conference and
Labs of the Evaluation Forum, CLEF ’2021, Bucharest, Romania (online), 2021.
URL: http://ceur-ws.org/Vol-2936/paper-30.pdf.
[2] P. Nakov, A. Barrón-Cedeño, G. Da San Martino, F. Alam, J. M. Struß, T. Mandl,
R. Míguez, T. Caselli, M. Kutlu, W. Zaghouani, C. Li, S. Shaar, G. K. Shahi,
H. Mubarak, A. Nikolov, N. Babulkov, Y. S. Kartal, J. Beltrán, M. Wiegand,
M. Siegel, J. Köhler, Overview of the CLEF-2022 CheckThat! lab on fighting the
COVID-19 infodemic and fake news detection, in: A. Barrón-Cedeño, G. Da San Mar-
tino, M. Degli Esposti, F. Sebastiani, C. Macdonald, G. Pasi, A. Hanbury, M. Pot-
thast, G. Faggioli, F. Nicola (Eds.), Proceedings of the 13th International Conference
of the CLEF Association: Information Access Evaluation meets Multilinguality,
Multimodality, and Visualization, CLEF ’2022, Bologna, Italy, 2022.
[3] P. Nakov, A. Barrón-Cedeño, G. Da San Martino, F. Alam, R. Míguez, T. Caselli,
M. Kutlu, W. Zaghouani, C. Li, S. Shaar, H. Mubarak, A. Nikolov, Y. S. Kartal,
J. Beltrán, Overview of the CLEF-2022 CheckThat! lab task 1 on identifying relevant
claims in tweets, in: N. Faggioli, Guglielmo andd Ferro, A. Hanbury, M. Potthast
(Eds.), Working Notes of CLEF 2022—Conference and Labs of the Evaluation Forum,
CLEF ’2022, Bologna, Italy, 2022.
[4] S. Shaar, M. Hasanain, B. Hamdan, Z. S. Ali, F. Haouari, M. K. Alex Nikolov,
F. A. Yavuz Selim Kartal, G. Da San Martino, A. Barrón-Cedeño, R. Míguez,
T. Elsayed, P. Nakov, Overview of the CLEF-2021 CheckThat! lab task 1 on
check-worthiness estimation in tweets and political debates, in: Working Notes of
CLEF 2021—Conference and Labs of the Evaluation Forum, CLEF ’2021, Bucharest,
Romania (online), 2021. URL: http://ceur-ws.org/Vol-2936/paper-28.pdf.
[5] P. Nakov, G. Da San Martino, T. Elsayed, A. Barrón-Cedeño, R. Míguez, S. Shaar,
F. Alam, F. Haouari, M. Hasanain, W. Mansour, B. Hamdan, Z. S. Ali, N. Babulkov,
A. Nikolov, G. K. Shahi, J. M. StruSS, T. Mandl, M. Kutlu, Y. S. Kartal, Overview
of the CLEF-2021 CheckThat! lab on detecting check-worthy claims, previously fact-
checked claims, and fake news, in: Proceedings of the 12th International Conference
of the CLEF Association: Information Access Evaluation Meets Multiliguality,
Multimodality, and Visualization, CLEF ’2021, Bucharest, Romania (online), 2021.
URL: https://link.springer.com/chapter/10.1007/978-3-030-85251-1_19.
[6] J. R. Martinez-Rico, J. Martínez-Romo, L. Araujo, Nlp&ir@uned at checkthat! 2021:
Check-worthiness estimation and fake news detection using transformer models, in:
CLEF, 2021.
[7] X. Zhou, B. Wu, P. Fung, Fight for 4230 at checkthat! 2021: Domain-specific
preprocessing and pretrained model for ranking claims by check-worthiness, in:
G. Faggioli, N. Ferro, A. Joly, M. Maistro, F. Piroi (Eds.), Proceedings of the
Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum,
Bucharest, Romania, September 21st - to - 24th, 2021, volume 2936 of CEUR
Workshop Proceedings, CEUR-WS.org, 2021, pp. 681–692. URL: http://ceur-ws.org/
Vol-2936/paper-57.pdf.
[8] I. B. Schlicht, A. F. M. de Paula, P. Rosso, UPV at checkthat! mitigating cultural
differences for identifying multilingual check-worthy claims, CoRR abs/2109.09232
(2021) 2021. URL: https://arxiv.org/abs/2109.09232. a r X i v : 2 1 0 9 . 0 9 2 3 2 .
[9] J. M. Pérez, J. C. Giudici, F. Luque, pysentimiento: A python toolkit for sentiment
analysis and socialnlp tasks, 2021. a r X i v : 2 1 0 6 . 0 9 4 6 2 .
[10] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional
transformers for language understanding, CoRR abs/1810.04805 (2018). URL:
http://arxiv.org/abs/1810.04805. a r X i v : 1 8 1 0 . 0 4 8 0 5 .
[11] D. Q. Nguyen, T. Vu, A. T. Nguyen, BERTweet: A pre-trained language model for
English Tweets, in: Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations, 2020, pp. 9–14.
[12] F. Barbieri, J. Camacho-Collados, L. Espinosa-Anke, L. Neves, TweetEval:Unified
Benchmark and Comparative Evaluation for Tweet Classification, in: Proceedings
of Findings of EMNLP, 2020.
[13] D. Croce, G. Castellucci, R. Basili, GAN-BERT: Generative adversarial learning
for robust text classification with a bunch of labeled examples, in: Proceedings
of the 58th Annual Meeting of the Association for Computational Linguistics,
Association for Computational Linguistics, Online, 2020, pp. 2114–2119. URL:
https://www.aclweb.org/anthology/2020.acl-main.191.
[14] D. Kartsaklis, I. Fan, R. Yeung, A. Pearson, R. Lorenz, A. Toumi, G. de Felice,
K. Meichanetzidis, S. Clark, B. Coecke, lambeq: An efficient high-level python
library for quantum nlp, arXiv preprint arXiv:2110.04236 (2021).