This paper describes the system proposed by the SATLab for hate speech and ofensive content identiifcation in five low-ressource languages. This language-agnostic system applies a classical supervised learning to character n-grams, using no other data than the learning materials. After optimizing a series of parameters, it ranked first in the Bodo task and second in the Gujarati task, for which the learning material contained only 200 tweets. It also performed well in the Sinhala and Assamese task, but was outperformed by several systems in the Bengali task.
This year, the SATLab team took part in five tasks proposed by HASOC 2023, the fifth edition
of the challenge on Hate Speech and Ofensive Content Identification in English and
IndoAryan Languages [
In this NLP field as in many other NLP domains, deep learning and pre-computed embeddings
are the preferred solutions, even in low-resource languages [
A first important feature of this 2023 edition of HASOC is that it includes languages that have never been the subject of this type of challenge, such as Sinhala, Bengali, Bodo and Assamese. The proposed system achieved excellent results for Bodo, finishing first, and for Sinhala, fourth, but very close to the best teams. A second important feature of HASOC 2023 is that for one of the languages, Gujarati, the learning material contained just 200 tweets. The organizers expected participants to explore various techniques to improve the system in a few short settings. The system developed by the SATLab does not use any information sources other than the learning material. Nevertheless, it achieved an excellent second place for this subtask, with a Marcro-F1 of 0.8383, 0.0105 points behind the best team.
The remaining of this paper presents the five tasks in which the SATLab took part and the challenge rules. Next, the general characteristics of the proposed systems are described. Finally, the results obtained are discussed.
All the tasks involved identifying hate and ofensive content in short messages posted on the Internet, such as tweets or YouTube comments. For each of them, the system was asked to distinguish between messages that included ofensive language such as insulting, hurtful, derogatory, or obscene content (HOF) and messages that did not (NOT). All the task languages in which the SATLab participated were poorly endowed with linguistic resources.
In Task 1A ([
Task 4 [
During the test phase of the challenge, five runs could be submitted for tasks 1A and 1B, while for three subtasks of Task 4, five runs could be submitted every day for more than a fortnight. The measure used to evaluate the systems is the Macro-F1 score.
The systems proposed for the five tasks are all derived from the system that achieved excellent
results in the 2021 and 2022 editions of HASOC. These were supervised approaches based only
on the learning materials provided by the task organizers. Two supervised procedures were
used: the LIBLinear L2-regularized logistic regression model (dual, -s 7) for classification [
Of the many parameters evaluated for the 2021 and 2022 editions of HASOC, the following two
were retained:
• The maximum length of ngrams, which could vary from 4 to 7. In all cases, all ngrams
shorter than this maximum value were used.
• The weighting applied to the frequency of each feature in an instance: the sublinear TfIdf
and BM25 [
In all the systems developed, the minimum frequency of a feature in the material analyzed has been set at 2, and the weighting scheme applied to all features in an instance is the L2 normalization.
For the LIBLinear L2-regularized logistic regression model (dual, -s 7), three parameters were evaluated: • The regularization parameter C. • The -w1 options for adjusting the parameter C of the HOF category.
• The bias parameter (-B), which shifts the separating hyperplane from the origin.
LightGBM’s parameters are far too numerous to present here. They have been optimized by the
automatic procedure described in Bestgen [
The parameters presented above were first optimized on the training material using a 4-fold cross-validation procedure stratified by category. Secondly, some trials allowed by the challenge rules were used to try to optimize these parameters for the test set. For each task, both the LIBLinear and LightGBM procedures were evaluated. In the challenge submissions, the whole training material was used.
This section successively presents the results of the systems submitted for each of the five tasks.
The cross-validation procedure on the training material led to the choice of the following parameters for feature extraction: maximum length of 5 characters and sublinear TfIdf. The cross-validation did not reveal any significant diferences between the supervised learning procedures and so both approaches were evaluated on the test material. The system that performed best was an ensemble of three other models: a LIBLinear (C=8, w1=1.8 and B=0.2) and two LightGBM models, the first based on the same features as the LIBLinear and the second based on ngrams ranging from 1 to 7 characters. These three models obtained cross-validation Macro-F1 scores of 0.8018, 0.8289 and 0.8295 respectively. The best of them obtained 0.8304 on the test material. The set of three systems (majority vote) came fourth in the challenge with a Macro-F1 of 0.8351, just 0.0031 behind the best team, as shown in Table 1.
As a reminder, the training set for this task contained only 200 instances. The cross-validation procedure on this set led to the choice of the following parameters for feature extraction: maximum length of 4 characters and sublinear TfIdf. Cross-validation showed that LightGBM (Macro-F1 = 0.75) was clearly more eficient than LIBlinear (Macro-F1 = 0.68, C = 3.5, B = 1). This observation was confirmed on the test material (but to a lesser extent) with Macro-F1 values of 0.8188 and 0.8130 respectively.
However, while precision and recall for the LightGBM version were almost identical,
LIBLinear’s precision (0.8890) was significantly higher than recall (0.7840), suggesting that the
system was assigning too few instances to the HOF category. On the basis of the probabilities of
belonging to this category returned by LIBLinear, the proportion of HOFs in the prediction was
increased by assigning to this category all instances with a probability greater than or equal
to 0.43 (instead of the default value of 0.50), raising the proportion of HOFs in predictions on
the test material from 0.20 to 0.28. This simple trick, which increased recall to 0.83 while only
reducing precision to 0.85, enabled the system to gain 0.02 points and take second place in the
challenge with a Macro-F1 of 0.8383, 0.0105 points behind the best team and more than 0.04
ahead of the third-placed team (see Table 1) . It would be interesting to compare the performance
of these systems using the bootstrap confidence intervals [
The proposed approach, which uses only 200 tweets for training, is therefore very efective. However, there is a significant and unexpected diference between the performance on the training material with a maximum Macro-F1 of 0.75 in cross-validation and the performance on the test material with a Macro-F1 of 0.84. This diference may be due to the fact that cross-validation training is carried out on only 150 tweets, whereas the test phase uses 200.
The results for these three subtasks are presented together because the proposed systems are very similar. The LIBLinear procedure is used in each case. Table 2 shows the parameters derived from the cross-validation and the Macro-F1 achieved on using the CV and on the test set. These systems ranked first for Bodo with a 0.006 lead over the 2nd team, fourth for Assamese with a 0.019 diference from the first and ninth for Bengali with a 0.10 diference from the first.
The SATLab approach for identifying ofensive content in short social network posts proved
highly efective for four of the five languages (Bodo, Gujarati, Sinhala and Assamese), but much
less so for the last one (Bengali), since the diference with the best team for these two languages
is almost 0.10 Macro-F1 score. The origin of these diferences is unknown to me. Only a reading
of the organizers’ synthesis [
The eficiency obtained for Gujarati is quite astonishing and unexpected for an approach that employs no other resources than the learning material, which is limited for this language to 200 instances. This result suggests that it would be interesting to repeat all the HASOC tasks proposed over the last five years and determine for each of them the impact of the number of instances available for learning on performance in the test phase. To be honest, I doubt that such good results could be obtained for all of them. The diference in performance in Task 4 between Bodo and the other two languages also merits further analysis.
The author wishes to thank the organizers of this shared task for putting together this valuable event. He is a Research Associate of the Fonds de la Recherche Scientifique - FNRS (Fédération Wallonie Bruxelles de Belgique). Computational resources have been provided by the supercomputing facilities of the Université Catholique de Louvain (CISM/UCL) and the Consortium des Equipements de Calcul Intensif en Fédération Wallonie Bruxelles (CECI).