We address the two subtasks of the Disaster-Related Social Media Data (DisasterMM) Task at MediaEval 2022: flood relevance classification and location extraction for Italian tweets. We demonstrate the ability of BERT-based deep learning models to address these subtasks. We then carry out error analysis, with an emphasis on the location extraction task, and find that our models (1) perform well with limited training data, (2) are robust in handling text with punctuation and special characters, and (3) are capable of detecting phrasal locations that can be expressed in diferent ways and can be very challenging for conventional gazetteer-based approaches.
With the advent of pre-trained language models such as BERT, once a suficient amount of
training data is available for specific NLP tasks, fine-tuning is often employed to address these
tasks successfully [
For RCTP, we employed diferent preprocessing techniques on the RCTP development dataset like stopword removal and stemming. However, best results were obtained with ‘Clean text’ consisting of lowercased tweet data, removing of whitespace, usernames, URL’s and retweets. Upon removing duplicate tweets and retweets, 2,524 tweets remained, which were then split in 80%-20% training-validation via stratified sampling. We ran Logistic Regression, Support Vector Machine (SVM) and Naive Bayes (NB) algorithms with default parameters on the diferent preprocessed text data.
For the training of our BERT models, we used the Transformer Toolkit [
As the automatic recognition of locations is one of the subtasks of Named Entity Recognition
(NER), it is an obvious choice to evaluate the efectiveness of existing NER tools for this specific
task of recognizing Italian locations mentioned in flood-related tweets. We did not employ any
text preprocessing techniques on the LETT development dataset. We did a train-validation
stratified sampling split of 80%-20%, resulting in 3,146 tweets as training set and 787 tweets
as validation set. We fine-tuned 4 diferent BERT-base models: BERT-base Multilingual Cased,
BERT-base Italian Cased and pretrained NER models called BERT-base NER and WikiNEuRal
Mulitilingual NER, all using the same 3,146 tweets train set. We selected the cased models to
better recognize locations that are proper nouns such as cities, towns, provinces, etc. BERT-base
NER is ready to use for NER, trained to recognize four types of entities and fine-tuned on the
English version of the standard CoNLL-2003 NER dataset [
On Punctuations and Special Characters We observed that all our BERT-base models are very robust against punctuations and special characters (periods, commas, hashtags, colons, ellipses, and others), which is useful as the tweet text format is unstructured. Although the BERT-base Italian Cased model had the lowest error rate at 0.038 when considering all special characters, it has a high error rate on other special characters (- ! " ()) at 0.142. Hence, we select the WikiNEuRal Multilingual NER having an error rate of 0.04 as the better choice when we want to catch all locations with special characters.
Learning from limited training We can confirm that our models were able to generalize street names when we investigated into the training data. Not only can our models capture street name tokens such as "#SS340", "A51", and "SS36", it was also able to predict these streets as "B-LOC" of phrasal locations. Here we show an example: (1)
SS36 Del Lago Di Como E Dello Spluga
B-LOC I-LOC I-LOC I-LOC I-LOC I-LOC I-LOC I-LOC However, there is still room for improvement. "Faro#Fiumicino," was annotated as "B-LOC" in the dataset, but were annotated as "O" by all the models. "Faro#Fiumicino," seems to be a flight route from Faro in Portugal to Fiumicino in Italy, making this specific token ambiguous. Our models were also not able to detect a very specific long location phrase "Tronco Maestro del Bacchiglione" which is a branch of the river Bacchiglione. Instead, all models predicted only the word "Bacchiglione" as a location.
Going beyond the gazetteer We want to confirm whether deep learning can detect locations above and beyond what can be detected by a simple, transparent, gazetteer-based approach. Our error analysis revealed that one place in which BERT has an advantage over gazetteer-based approaches is locations that consist of multiple words (“phrasal locations”), which are highly variable and would be dificult to find in a gazetteer or include in a fixed vocabulary. Take for example "Milano". In our validation set, it appears 61% of the time as "B-LOC" and 33% as "I-LOC" (ínside a phrasal location) and 6% as "O". WikiNEuRaL NER and BERT-based Multilingual predicted all of these instances correctly.
We are particularly interested in considering the fit between the ground truth of the DisasterMM task and possible real-world applications of the technology for disaster management. Here, we mention two aspects of the ground truth that we discovered while carrying out the task that would be interesting to address in the future.
First, we observed some inconsistencies in the annotations. For example, we observed in the ground truth that "Toscana" appeared 69 times in the validation set, 48 annotated as "B-LOC" (location) and 21 annotated as "O" (not location). Examples of the mismatches are: Both occurrences of Toscana were annotated as Not Location: Allerta meteo Toscana - 05/04/2017 12:00 - Regione Toscana - https://t.co/NwsxgzAAiK https://t.co/6Rli3ttdfs Both occurrences of Toscana were annotated as Location: Allerta meteo Toscana 14/06/2017 02:00 - Regione Toscana - https://t.co/NwsxgzAAiK https://t.co/VrQtOuYV7J We infer that these were annotation errors: "Toscana" should be location. All of our models tag "Toscana" as "B-LOC" in all instances, meaning that some of the errors our model makes might not be actual errors. Other possible inconsistencies are "Bacchiglione...", "vicentino:", "Roma." and "Incrocio" (Eng. “intersection”).
Second, we observe that there are some location-related entities in the data set that are not annotated as locations, but would be important in the context of flood disaster management. Specifically, flooding events will often refer to rivers, and valleys crucial locations, but do not appear to be included in the definition of the location used to annotate the data. These events do not have a single location in terms of latitude and longitude. However, there is no principled reason for which they cannot be considered locations, since roads are annotated as locations in the dataset. The conventional datasets used to train NER tools are based on standard news articles that usually refer to cities and country names. Future work should focus on designing annotation protocols with experts that annotate all location- related entities that could possibly be relevant for flooding. Looking towards real world applications, we intend to improve our methodology to better detect specific phrasal locations for decision support in disaster management.
Acknowledgments This work is partly financed by the Dutch Research Council (NWO) with project number NWA.1292.19.399.