1. Introduction
of a message.
The potential of language models has made them
The most likely source of the vertiginous progress of the baseline of a wide range of NLP tasks, and they
Natural Language Processing (NLP) in the recent can even be used for developing learning models in
years is the proposal of the Word2Vec model [
1
], production environments. On the other hand, the
which eases the generation of unsupervised linguis- ease of tuning these models to specific NLP tasks has
tic features that are known as word embeddings led the development and release of a huge amount
and they represent the meaning of words in vec- of pre-trained language models in a large bunch
tors of real numbers. The strong results reached of NLP task, with HuggingFace and especially its
by word embeddings based on Word2Vect enhanced Transformers library [
4
] standing out. This vast
the design of new word embeddings models, such variety of language models makes their comparison
as Glove.1 These models set an embedding vec- and analysis really dificult as a previous step of the
tor to each word regardless of its context, and for particular language model to fine-tune to a specific
this reason the next landmark were starred by the use case.
contextual word embeddings models [
2
]. The trans- The certain use of language in social networks
formers models stand out as contextual word em- makes to adapt the NLP methods to the specific use
beddings, with BERT [
3
] as outstanding example. of language of each social network, as for instance
These models are known as language models, and to Twitter [
5
]. Language models also needs this
their capacity of representing the meaning of words iftting to the use of language of social networks,
couple with the possibility of using them as pre- which makes them to be at the top of most NLP
trained models have driven the progress of a broad shared-tasks.
branch of NLP tasks, especially those mostly linked The great availability of language models has
to the classification of the semantic meaning of text, not been coupled with the release of web platforms
such as the opinion polarity of a review, the ofen- for comparing and analysing the diferent language
sive meaning or the underlying emotional meaning models in specific NLP tasks. Nevertheless, the
issue of the great availability of training corpora
and the evaluation of learning models begins to
be resolved by the publication of leader boards of
learning models trained on gold standards, such as
SuperGLUE [
6
] or TweetEval [
7
].
Following the example of the NLP classification
tasks leader boards, we present the web platform
NLP4SM,2,3 whose demonstrative prototype is
described in this paper. NLP4SM is a web application
for analysing the performance of Twitter language
2Prototype: https://nlp4sm.on.fleek.co/
3Production [
8
]: https://tweetnlp.org/demo/
models fine-tuned to the tasks of (1) sentiment anal- Irony detection The goal is to classify whether a
ysis, (2) emotion analysis, (3) ofensive language tweet is ironic. The corpus of the Irony Detection
classification, (4) hate speech classification, (5) irony task from SemEval18 was used to fit the model
detection and (6) stance classification on abortion, [12].7
climate change, atheism, feminism and Hillary
Clinton. NLP4SM allows on one hand the classification Ofensive language It identifies whether a span
of a free span of text, and on the other hand the of text has an ofensive meaning. The corpus of
classification of the meaning of a bunch of tweets OfensEval from SemEval19 was used to fit the
returned by Twitter. Furthermore, the classification model [13].8
results are shown as charts to ease their
understanding. NLP4SM can be used by non-NLP experts and Emoji prediction It aims at predicting the emoji
NLP scientists that need to compare diferent lan- that best represent the meaning of a tweet. The
guage models in one of the mentioned tasks on real corpus of Emoji Prediction from SemEval18 was
data. The design of the system allows the consider- used to fit the model [14]. 9
ation of new language models of the previous NLP
tasks, as well as the incorporation of new result
visualisation methods.
2. Language Models in NLP4SM
The first version of NLP4SM incorporates learning
models that classify the meaning of tweets. The
learning models are based on the fine-tuning of
Twitter language models to the specific NLP tasks,
which we subsequently describe.
2.1. NLP tasks
We select the NLP tasks according to their scientific
relevancy, as well as the high social demand to have
automatic systems that can identify specific kind
of messages. The tasks are also part of TweetEval,
and we present them as what follows.
Stance classification It classifies the author stance
according to a topic. The corpus of the task Detectin
Stance from SemEval16 was used to fit the model.
The topics considered are: abortion,10 atheism,11
feminism,12 climate change13 and Hillary Clinton.14
Multinguality Social networks are multilingual,
and for this reason NLP4SM also allows to analyse
multilingual language models, namely those ones
based on XLM-R [15] that is fitted on a large set of
tweets written in more than 50 languages. NLP4SM
also provides the XLM-T language model fitted to
the sentiment analysis task in eight diferent
languages [16].
2.2. Language Models
The language models currently included in NLP4SM
match with the ones in TweetEval and they are
availEmotion analysis It identifies the underling emo- able in HuggingFace. We have used the
RoBERTation of a text. Although it is a multi-label task, we base model [17] pre-trained on English text from
redefined it as a multi-class classification task. The social networks [
7
].
corpus “Afect in Tweets” [
9
] was used to fit the The fine-tuning of RoBERTa-base to each NLP
model to the most frequent emotions of the corpus: task is based on a output layer with the same output
joy, optimism, anger and sadness.4 units than the number of classes of each task [17].
Sentiment analysis It classifies the opinion polar- 7https://huggingface.co/cardifnlp/
ity in positive, negative or neutral. The corpus of twitter-roberta-base-irony
the subtask A of “Sentiment Analysis in Twitter” 8https://huggingface.co/cardifnlp/
task of SemEval17 [
10
] was used to fit the model. 5 twitter-roberta-base-ofensive
9https://huggingface.co/cardifnlp/
twitter-roberta-base-emoji
Hate speech It aims at classifying whether a tweet 10https://huggingface.co/cardifnlp/
express hate. The corpus of HateEval from Se- twitter-roberta-base-stance-abortion
mEval19 was used to fit the model [
11
]. 6 11https://huggingface.co/cardifnlp/
twitter-roberta-base-stance-atheism
4https://huggingface.co/cardifnlp/ 12https://huggingface.co/cardifnlp/
twitter-roberta-base-emotion twitter-roberta-base-stance-feminist
5https://huggingface.co/cardifnlp/ 13https://huggingface.co/cardifnlp/
twitter-roberta-base-sentiment twitter-roberta-base-stance-climate
6https://huggingface.co/cardifnlp/ 14https://huggingface.co/cardifnlp/
twitter-roberta-base-hate twitter-roberta-base-stance-hillary
4. Conclusions and future work
The languages models used are described and linked in section 2.1.
In this paper, we presented the prototype demon
stration NLP4SM, which aims at easing the access,
3. Description of NLP4SM analysis and comparison of classification models
based on language models of diferent NLP tasks
We aim at providing an unified and accessible plat- with real data from social networks. NLP4SM
alform for assessing and analysing social network text lows the evaluation of any span of text, and the
classification models. Hence, we have developed a evaluation of tweets from a user query.
web application for the first version of NLP4SM. We plan as future work: (1) to integrate more
NLP4SM is built upon a client-server architecture NLP tasks, (2) to extend the number of language
led by a REST API. Moreover, we have relied on models considered, and (3) to add a greater number
external services for running the language models. of visualisation methods of results.
NLP4SM uses Huggingface because it is currently
the on-cloud service that hosts the language models
included in NLP4SM, it is the artificial intelligence Acknowledgments
service platform most used by the NLP research
community and it provides a high quality service. This research work is supported by the R&D&I
The server side is developed in Python and it is grant PID2020-116118GA-I00 funded by MCIN/
based on the micro-framework Flask. The server AEI/10.13039/501100011033.
side is responsible of the communication with
HuggingFace through using its API. Moreover, the References
server side queries Twitter according to the user
query.
The client side is a web interface based on
JavaScript React. It allows two diferent forms
of evaluating the models, namely:
Text mode It evaluates any language model de
scribed in section 2 with a span of text written
down by the user in a text box. Several charts show
the result of the evaluation. Figure 1 depicts and
example of the text mode.
Twitter mode It process a set of tweets returned
in real-time from Twitter according to the user
query. The user can configure his query according
to the language, the time and the specific text of
the query. NLP4SM retrieves the tweets and shows
with diferent kind of charts the result of running
the selected language model. Figure 2 depicts and
example of the text mode.