Text classification is a typical and important part of supervised learning, it has several applications in economics and attracted the attention of many stock market investors. For a long time, the news is frequently an unanticipated stock investment variable that instantaneously influences stock price directions. In front of an enormous volume of news, investors are always searching for models that automatically categorize news quickly and accurately. Thus, in this research, we have utilized diferent models like PhoBERT, SVM, Logistic Regression, LSTM, Random Forest, and Naive Bayes to classify news articles into three categories [negative, neutral, or positive] based on their titles. The results demonstrated that after training with a dataset of over 1000 news samples from CafeF.vn, the PhoBERT model outperformed other models with an accuracy up to 93%. The code and dataset is available at https://github.com/209sontung/Vietnamese-stock-article-classification.
Text classification is a traditional word processing problem using machine
learning. The first idea is to map a text to a known topic from a finite set of topics
based on the semantics of the text. Text documents are typically used for
classiifcation, which is done based on selected documents and features. However, the
classes are chosen before the experiment analysis, which is referred to as
supervised machine learning operations. Due to the growing number of documents,
the demand for text classification is expanding, and the tasks are getting
increasingly diverse, such as sentiment analysis of reviews and news categorization, so
on. An article in a newspaper, for example, could fall under one (or more) of
these categories (such as sports, health, information technology, etc.). The use of
spam filtering in email [
Forecasting is always a dificult task in the stock market because it is highly volatile and dynamic. Many methods have been proposed to forecast the future direction of the stock market. Financial news, for instance, has an immediate positive or negative impact on stock prices. Before purchasing a stock, investors, for instance, evaluate a company based on its activities on its oficial website and ifnancial news about the company. However, investors can not fully assess such vast amounts of financial news data on their own. As a result, investors require a model that can assist them in quickly sorting through financial news articles.
In this research, we collected a dataset of over 1000 financial news
articles about the stock market from the website CafeF.vn. Afterward, we used
LSTM [
The remainder of this paper is organized as follows. Section 2 introduces related works . The proposed model is introduced in Section 3. Section 4 discusses the results of experiments and is followed by a conclusion in Section 5. 2
Sentiment analysis, also referred to as a classification task, aims to forecast the overall sentiment of a text, which could be a tweet or a review of a movie or product. The main goal is to determine if the text’s conveyed impression is positive or negative, in some cases, with a score or confidence metric.
In English, a lot of publications on sentiment analysis have been undertaken.
For the problem of sentiment classification, Pang et al compared multiple
supervised learning algorithms [
In Vietnamese, Kieu and Pham [
In this paper, we proposed a Vietnamese stock news sentiment classification model, which is a novel approach to sentiment analysis in Vietnamese. The proposed model achieved 93.12% accuracy and was constructed using a pretrained PhoBERT, a state-of-the-art language model for Vietnamese based on BERT architecture. In addition, we built a dataset that included 1000 titles of financial articles taken from CafeF.vn and labeled them into three groups [negative, neutral, or positive]. 3
Our proposed model consists of 2 stages described in Fig. 1. The input is the ifnancial article’s headlines, which will proceed through the first stage to preprocess the data to convert it into a format that the PhoBERT model can understand and improve its accuracy. Following that, in the second stage, our PhoBERT-based model will be tasked with assessing content from the header broadcast and categorizing it into one of three classes represented as -1, 0, or 1 (i.e., -1 as negative, 0 as neutral, and 1 as positive direction).
The preprocessing procedure was separated into two phases. In Phase 1, first, we
applied VnCoreNLP’s Named entity recognition [
In Phase 2, we had the symbol vocabulary with the character vocabulary, and each word was represented as a sequence of characters with a unique end-ofword symbol ” < /s > ” that allowed us to recover the original tokenization after translation. In example, we counted all symbol pairs iteratively and replaced each occurrence of the most common pair (”A”, ”B”) with the new symbol ”AB”. Each merge process generates a new symbol that represents an n-gram of characters. BPE does not require a shortlist because frequently occurring character n-grams (or complete words) are finally combined into a single symbol. Thus, the amount of the final symbol vocabulary is equal to the original vocabulary.
Then we mapped each subword to its corresponding ID in the PhoBERT vocabulary, and because each title is varied in length, we employed pad sequences to match them all in length. i.e. sentences that shorter than 125 subwords are padded with 0 at the end, while longer are trimmed to produce 125. 3.2
First, we installed all the necessary materials included transformers library, VnCoreNLP Python wrapper and its word segmentation component (i.e. RDRSegmenter), then fastBPE to convert the input text into a list of subwords. After train and test dataset are done prepared as described in Section 3.1, DataLoader is created to load data into the model. Then we loaded P hoBERTBASE from HuggingFace’s transformers library as the pre-trained model. The optimizer we chose for the training stage is AdamW optimizer, which is an improved version of Adam optimizer from the transformers library. In addition, this study used batch size = 32 with 10 epochs divided into two stages with diferent learning rate. The initially learning rate which we utilized was α = 5e − 6 in the first 5 epochs in order for the loss to converge faster. Once the first training phase was completed, we reduced the learning rate to α = 5e − 7 to achieved the smallest potential loss. The model was then trained for another 5 epochs since the loss on the validation dataset appears to stabilize after this number of cycles.
Finally, the softmax classicfiation layer (which includes three nodes corresponding to three classes in the dataset) will output the probabilities of the input text belonging to each of the class labels, with the total of the probabilities equal to 1. The dense layer consists of a fully connected neural network with the softmax activation function. The softmax function σ : RK in (1). → RK is given σ (z)i =
ezi
PK
j=1 ezj
f or i = 1, ..., K
(1)
where z = (z1,...,zK ) ∈ RK is the softmax layer’s intermediate output (also
called logits). The predicted label for the input is then chosen from the output
node with the highest likelihood. The output of the proposed model will be
represented as -1, 0, or 1. The entire training process was deployed on PyTorch
[
BERT. BERT stands for Bidirectional Encoder Representations from
Transformer. It is a transformer-based [
BERT has performed more than 10 natural language processing tasks with good results. It has improved the GLUE benchmark to 80.5%, pushed MultiNLI accuracy to 86.7%, absolute 5.1 point improvement in SQuAD v2.0 Test F1, etc. With L is the number of sub-layer blocks in the transformer, H: the size of the embedding vector, A: the number of heads in the multi-head layer, model BERT has two architectures as follows: – BERTBASE (L=12, H=768, A=12): Total parameters are 110 million. – BERTLARGE (L=24, H=1024, A=16): Total parameters are 340 million. PhoBERT. The BERT model’s release marked a watershed moment in the NLP industry. Following the public release of the BERT model, a slew of open-source BERT training programs have sprung up. There are also numerous unilingual and multilingual BERT pre-train models that are commonly used. Since then, PhoBERT has been particularly trained for Vietnamese and released by VinAI Research in March 2020.
PhoBERT is based on the design and approach of RoBERTa [
VnCoreNLP. VnCoreNLP (A Vietnamese Natural Language Processing Toolkit) is a Java Natural Language Processing toolkit designed to aid NLP research in Vietnam. Through essential NLP components such as word segmentation, POS tagging, and NER, VnCoreNLP provides extensive linguistic annotations. In the course of NLP research, the Vietnamese standard dataset was published. In early 2013, the first VLSP evaluation campaign used datasets for word segmentation and POS tagging. In 2014, a high-quality dependency treebank was published, and a NER dataset was published for the 2016 VLSP review campaign. The architectural system design is depicted in Fig. 2.
To be able to use PhoBERT to evaluate and categorize the news’ impact, we provided a dataset that included 1000 titles of financial articles taken from CafeF.vn and labeled them into three groups [negative, neutral, or positive] with the help of experts. The dataset contains 187 articles having a negative impact, 248 articles with no impact, and 565 articles with a positive impact. After that, we divided the dataset into three sets, 80% for training, 10% for validation and 10% for testing. The training set was used to train the model, validation set was utilized to tune the hyper-parameter. Finally, the result of model was evaluated on testing set. The examples of our dataset are shown in Table 1. 4.2
After experimenting with 6 diferent models on the same dataset using various preprocessing techniques in order to achieve the best possible results, Table 2 was obtained.
As seen in the table above, our model, PhoBERT, outperformed other popular and sophisticated NLP models with 93.12% accuracy and was 10.54% higher than the second-highest approach using Logistics Regression. Our model also achieved the best performances in other metrics such as precision, recall and F1 score. In this research, sentiment classification was performed on Vietnamese stock articles collected from the site CafeF.vn. The whole dataset consists of 1000 titles divided into positive, neutral, and negative news. Because of the fact that white space is also utilized to separate syllables that make up words in Vietnamese, we utilize Rdrsegmenter from VnCoreNLP (a word splitting library published by the author of PhoBERT) to separate words for input data. Then, using the BPE Encoder, convert text into a list of subwords, then map each subword to its ID in the PhoBERT vocabulary. The proposed model in this study employed a state-of-the-art language model for Vietnamese named PhoBERT, and the entire training process has been deployed on PyTorch. As a result, our approach achieved 93.12% accuracy, surpassing other popular and sophisticated NLP models when performed on the same dataset.
As previously stated, financial news can directly impact stock prices, so our proposed model could be used to assist with stock price forecasting problems using machine learning. In future work, we also want to explore the efect of using word embeddings on sentiment classification. Furthermore, we aim to investigate multiclass categorization of news data using various Deep Learning models, which will comprise classes such as the economy, sports, health, and technology. Also, we intend to extend sentiment classification to more domains by crawling data from numerous Vietnamese websites, such as product reviews, hotel reviews, and book reviews.