This paper proposes a Malay Part-of-Speech (POS) tagger using a new set of POS tags and a deep learning classifier. A new set of POS tags was proposed, and the POS classifier was built using a deep learning model, a bidirectional long short-term conditional random field (BiLSTM-CRF). We expanded the POS tagset by considering the informal Malay terms frequently used in Malay tweet text. Additionally, for the Bi-LTSMCRF model, we used a combined embedding of Malay Wiki Word2Vec and a Malay Twitter corpus annotated with POS Word2Vec. The BiLSTM-CRF Malay POS tagger was then compared to traditional classifier models. Based on evaluation results, the BiLSTM-CRF model performed the best. Thus, we concluded that deep learning techniques combined with appropriate embeddings are capable of performing fine-grained POS tagging on Malay tweet text. Additionally, using a customized POS tagset for specific types of text results in increased POS label coverage.
The Part-of-Speech (POS) classification method organizes words into groups based on how they are
used and function in sentences. A POS tagger, on the other hand, is a software component that reads a
text in many languages and assigns appropriate words to each word (or token) in the text. Malay has
four primary POS tags: Nouns, verbs, adjectives, and word tasks [
Additionally, Malaysians, particularly adolescents, are incredibly inventive when it comes to writing and coining new words [5]. As a result, Malay social media text, such as Malay tweets, is dense with colloquial Malay and peppered with mixed-language phrases and derogatory terms. As a result, POS tagging a Malay tweet is extremely complicated, time-consuming, and energy-intensive.
Obtaining the POS tag to label these informal terms is one of the challenges. Some researchers either ignore these terms (by failing to label them) or repurpose the existing POS tagset. Both approaches reduce the accuracy of POS tagging on tweet text and cause POS tags to be mislabeled. To address this issue, we propose a new set of POS tags customized for Malay social media text, the tweet. On the other
2022 Copyright for this paper by its authors. hand, the POS tagging problem is highly dependent on the preceding and following sequences. Applying a neural network algorithm, such as a bidirectional long short-term memory network (BiLSTM) model to POS tagging will be extremely beneficial. In other words, the architecture of the BiLSTM model, which considers both previous and current sequences, increases the likelihood of constructing a high-accuracy prediction of the POS tag on the Malay tweet. Thus, to develop a highperformance POS tagger for Malay tweets, we propose a new POS tagset and a BiLSTM with Conditional Random Field (CRF) or BiLSTM-CRF model.
Our work in this paper consists of two parts: (1) to propose a finer POS tagset that covers nearly all types of words in the tweet text. (2) to build a POS tag classifier model based on the BiLSTM-CRF model. The following will describe in detail of our work: 2.1.
Malay has four primary POS tags, but these are insufficient for categorizing words in the Malay
Twitter corpus. As a result, we chose to create new Malay POS tags through the use of Malay formulas
and the grammar of Safiah et al. [
As a result, we chose to use Le et al. [
For the POS tag classifier model, we constructed our Malay POS tagging model based on BiLSTMCRF. The model is based on these three phases: (1) data preparation, (2) building/training the BiLSTMCRF Malay POS model, and (3) model evaluation. 30% (538) of the total data was set as test data in the data preparation. When training the biLSTM-CRF Malay POS model, three kinds of layers were built: The embedding layer, the BiLTSM layer, and the CRF layer (see the Figure 1):
At the embedding layer, a combination of embeddings was proposed; the Word2Vec of Malay Wikipedia [11] and the Word2vec of an annotated POS Malay tweets corpus. Both of the embeddings were trained with the parameter setting of 300 dimensions. As for the BiLTSM layer, the BiLTSM acts as a word feature extraction. Bi-LSTM takes contextual information and identifies class tagging potentials for each input. Finally, the CRF layer is used to classify the POS of a word. The CRF score function is used to find the POS sequence with the highest score and calculate the probability distribution of all POS sequences. The CRF layer benefits from knowing what word should be followed by what word. Such features are difficult to demonstrate in the neural network layer, especially on small data sets [6]. However, with the help of the CRF layer, even a model trained with a small dataset can be used to classify things well.
Using the prepared dataset, we evaluated our BiLSTM-CRF model. In evaluating our model, a similar evaluation of Jason [12] that used training loss and validation loss were used to ensure our BiLSTM-CRF model was not underfit or overfit. Based on Figure 2 below, the small gap in learning curves between training loss and validation loss indicates the model was neither underfitted nor overfitted.
Afterwards, we ran several trainings using a set of hyperparameters. The most optimized hyperparameters for our BiLTSM-CRF Malay POS model were; learning algorithm Adam, dropout point 0, batch size 6, and 30 epochs. With such setting, our BiLSTM-CRF model managed to get 94% of Precision, Recall and F1-Score.
To evaluate our POS model, we compared it against four well-known traditional classifier models: Support Vector Machine (SVM)[7], Nave Bayes (NB)[9], Decision Tree (DT)[8] and K-nearest neighbour (KNN)[10]. All of the traditional models were trained based on word position features. To be precise, ten types of features were extracted: the preceding and following words; the prefix of each word (limited to the first three characters of the word); the suffix of each word (limited to the last three characters of the word); the length of the word; and the presence of a digit in the word. To train and evaluate traditional classifier models, the same dataset of Malay tweets (used in training and evaluate the BiLSTM-CRF mode) was used and divided into 70% (1,253 data) as trained data and 30% (538 data) as test data with 10-fold cross validation. To compare our BiLSTM-CRF model against the traditional classifier, evaluation metrics of precision, recall, and F1-score were used. The findings based on the metrics are shown in Table 1.
In conclusion, by creating new POS tags tailored to the words in Malay tweets, we ensure fewer unlabeled words with POS. The training of the BiLTSM-CRF model based on a combination of specific embeddings aids in increasing the performance in tagging POS in the tweet text. In other words, our proposed BiLSTM-CRF Malay POS tagging model and the new POS tagset are suitable for tagging POS in the Malay tweet text.
This project is supported by the Malaysia Ministry of Higher Education under research code:
FRGS/1/2020/ICT02/UKM/02/1
[
Bahasa dan Pustaka, Kuala Lumpur, 2010.
[
Lumpur, 2005.
[
Journal of Language Studies, 18 (2018) 124 -142. [5] N. Jamali, Fenomena Penggunaan Bahasa Slanga dalam Kalangan Remaja Felda di Gugusan Felda
Taib Andak: Suatu Tinjauan Sosiolinguistik, Jurnal Wacana Sarjana, 2 (2018) 1-1. [6] L. Marz, D., Trautmann, B. Roth, Domain Adaptation for Part-of-Speech Tagging of Noisy UserGenerated Text, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, 2019. [7] T. Nakagawa, T. Kudoh, Y. Matsumoto, 2001. Unknown Word Guessing and Part-of-Speech Tagging Using Support Vector Machines, in: Proceedings of the Sixth Natural Language Processing Pacific Rim Symposium, 2001, pp. 325--331. [8] L. Màrquez, H. Rodríguez, Part-of-speech tagging using decision trees, in: Proceedings of the
European Conference on Machine Learning, 1998, pp. 25-36.
[9] R. Creţulescu, A. David, D. Morariu, L. Vinţan, Part of speech tagging with Naïve Bayes methods,
in: Proceedings of the 18th International Conference on System Theory, Control and Computing
(ICSTCC), 2014, pp. 446-451.
[10] S. Gaber, N., Nazri A. M. Z., Omar, N., Abdullah, S., Part-of-Speech (POS) Tagger for
Malay Language using Naïve Bayes and K-Nearest Neighbor Model. International Journal
of Psychosocial Rehabilitation, 24 (2020) 5468-5476.
[11] Malaya toolkit, Natural Language Toolkit library for Bahasa Melayu, 2020. URL:
https://malaya.readthedocs.io/en/4.0/load-wordvector.html
[12] Y. Zhang, X. Wang, Z. Hou, J. Li, Clinical named entity recognition from Chinese electronic
health records via machine learning methods, JMIR Medical Informatics, 6 (2018) e50.
The proposed new POS tagset for Malay informal text which are the combination of the POS tagset from Safiah et al. [
POS tag