Fake news refers to a kind of public opinion that uses false information to deceive the parties in order to achieve a certain purpose. It fails to truly reflect the original appearance of objective things and contains false elements. The information provided by fake news is designed to manipulate people for different purposes: terrorism, political election, advertising, satire, etc. In social networks, fake news spreads in seconds among thousands of people and research has shown that misinformation spreads faster, farther, deeper, and more widely than true information [12], so it is necessary to develop tools to help control the amount of fake news on the network.
A few years ago, the method of detecting fake news was mainly to analyze the effective features from various sources, including the content of the text, user data and the form of news dissemination. It mainly distinguishes true and false news from the aspect of language features, such as writing style and special headlines [7], vocabulary and syntactic analysis [10]. In addition to language features [3], some studies have proposed classification schemes on user features and time features [7]. Recent fake news detection methods mainly use machine learning and deep learning techniques, with special attention to language-based methods [8,11,14,15]. Some people use TF-IDF feature extraction technology for fake news detection and have achieved good results [1,13,2].
A fake news detection system is designed to help user detect and filter potentially deceptive news. A predictive method of deliberately misleading news is based on the analysis of the real and deceptive previously censored news, that is, the annotated corpus. Text is main carrier of news information, and the study of news text helps to effectively identify fake news. The specific task is: given the text of a news event, determine whether the event is real news of fake news. For the evaluation of systems, we will use a new testing corpus containing news related to COVID-19 and news from other Ibero-American countries. Its availability will introduce two main challenges to the task: thematic and language variation. Our systems need to take into consideration that part of the testing corpus contains news in a topic that does not exist in training corpus, likewise, we should take into account that the other part of the testing corpus contains news in a different variation of the Spanish that is in training corpus. This paper proposes a method for fake news detection: A fake news detection method based on TF-IDF and ensemble machine learning.
TF-IDF has the characteristics of simplicity, fast calculation. and it performs well for processing long texts. The Section 2 introduces the corpus and analyzes the composition and distribution of the data. The third section introduce the methodology, data processing methods, feature extraction methods, the base model used, and the final ensemble model. The experiments sett and results are presented in Section 4. Finally, Section 5 outlines the final conclusions and future work.
The Spanish fake news corpus [9] is news collected from several online sources: existing newspaper websites, media company websites, special websites dedicated to verifying fake news, and websites designated by different reporters as regular fake news publications. All these articles are written in Mexican Spanish. The corpus collected 971 news items from different sources from January to July 2018: 971 news items were divided into training set and test set. Among them, there are 676 pieces of data in training set and 295 pieces of data in the test set. Only two categories (True of Fake) are considered for the marking of the corpus, and the specific conditions of each piece of data are as follows:
-Category: Fake/ True.
-Topic: Science/ Sport/ Economy/ Education/ Entertainment/ Politics, Health/ Security/ Society. -Headline: The title of the news.
-Text: The complete text of the news.
-Link: The URL where the news was published.
Among them, the number of fake news and real news is fairly balanced. And the number of fake news and real news in the Topic column is almost the same. However, there is a big gap between the authenticity of news published on different websites. Some websites are almost all fake news, and there are also websites that are all real news. This provides a good idea for our feature extraction. We will consider the impact of "Link" and "Topic" on the classification results when we do experiments.
This section includes 4 parts: data preprocessing, feature extraction methods, classification models, and ensemble model methods.
In order to get better results, data preprocessing is essential. And data preprocessing is usually the first step in natural language processing tasks. First intercept the most critical information in Link, and the result after interception looks like this: www.elruinaversal.com. Because we have analyzed the sources of fake news and found that almost all the news on some websites is fake news. Then we observe the data and find that each row consists of Category, Topic, Source, Headline, Text and Link. All the contents in Category, Topic, Source, Headline, Text and Link are merged as our new input. Finally, data cleaning is performed on the merged input data. Perform data cleaning on the merged data, use regular expressions to remove links, special symbols, punctuation, etc. According to the length of the text, the length of the longest text is 2578 and the shortest text is 31. So we decided to use nltk to remove the stop words in the text. The longest text length after removing the stop words is 1379, and the shortest is 18. Removing stop words in the text will reduce the effect of the model, and not removing stop words will improve the performance of the model. However, we have proved through experiments that removing stop words will reduce the performance of the model, which will be explained in subsequent experiments. And we will verify it in the next experiment. The data processing is: 1) merge all columns + remove stop words: 2) merge all columns + without remove stop words: 3) only Text + remove stop words: 4) only Text + without remove stop words.
The method base on news content focuses on extracting various features of fake news content, including knowledge-base and style-based features. This paper mainly uses two methods to extract text features: 1) LabelEncoder; 2) TF-IDF.
-LabelEncoder: We use Sklearn's LabelEncoder method to hard-code text features, that is, to encode discrete numbers or text, and convert the discrete data to numbers between (0, n-1), where n represents different data values. We performed LabelEncoder on the Topic and Source features. The LabelEncoder method also played a very good role in the experiment. -TF-IDF: Term Frequency-inverse Document Frequency (TF-IDF) is a statistical analysis method for keywords, used to evaluate the importance of a word to a document set or a corpus. The importance of a word is positively correlated with the number of times it appears in the article, and negatively correlated with the number of times it appears in the corpus. TF-IDF can effectively avoid the influence of commonly used words on keywords and improve the relevance between keywords and articles. TF refers to the total number of times a word appears in the article. This indicator is usually normalized and defined as the number of times a word appears in the article divided by the total number of words in the article, which can prevent the result from being biased towards too long document (The same word usually has a higher word frequency in a long document than in a short document). IDF refers to the frequency of reverse documents. The fewer documents that contain a word, the greater of the IDF value, indicating that the word has a strong ability to distinguish. Using TF-IDF can well extract text features in Spanish news. For texts with a length of several thousand, TF-IDF is better than RNN and other neural network model in extracting features of long texts. And for the challenge of changing language, TF-IDF can easily solve it.
We use five basic weak classifiers as our base model.
-
LightGBM [6]: The model is a fast, distributed, high-performance gradient boosting framework base on decision tree algorithm. It can be used in sorting, classification, regression, and many other machine learning tasks. LightGBM is an improvement of the GBDT algorithm [4]. LightGBM didn't use the traditional pre-sorting idea, but optimizes the histogram of the eigenvalues. The weak classifier is used to iteratively train to obtain the optimal model, which has the advantages of good training effect and not easy to overfit. The train method is GBDT: it is an algorithm that classifies or regresses data by using an additive model and continuously reducing the residuals generated during the training process. Choose the above LogisticRegression (LR), SGDClassifier (SGDC), PassiveAg-gressiveClassifier (PAC), RidgeClassifier (RC), LnearSVC (LSVC) as the weak base model. Figure 1 shows the use of the Stacking method in the ensemble learning method to predict all the trained base models on the entire training set, and each base model will get a classification prediction result. For each base model, we train out model by using 5-fold cross-validation, concatenate the classification prediction results after each base model training, and finally send all the features to the final LightGBM model for training.
First, data processing is performed, and then all the hyperparameter of the experiment are introduced. Almost all base model use default parameters, and each base model uses 5-fold cross-validation for training; the LightGBM model also uses 5-fold cross-validation for training. The training model of the LightGBM model is GBDT; the learning rate is set to 0.01; the maximum number of iterations num-boost-round is set to 10000; the progress is displayed every 50 iterations. Finally, the model outputs the final accuracy rate and and F1-macro. The hyperparameters of each classifier are as follows: LR (random-state=1017, C=3), SGDC (random-state=1017, loss='log'), PAC (random-state=1017, C=2), RC (random-state=1017), LSVC (random-state=1017).
We will evaluate the model from F1 measure and accuracy on the "fake" class. The results of the base model and ensemble model on the training data shown in Table 1, where "Merge-All" is to concatenate all the information, and "Stopwords" is removing stop words from the text. Accuracy and F1-macro are both results obtained on the validation set. From Table 1, we can see that the per- We can get that in the same base model, merge can play a certain role in improving the accuracy of the model and the F1-macro score. Without the merge, the accuracy of the model is significantly reduced, and the remove stop words from the text will reduce the performance of the model. At the same time, no matter how the data is processed, the ensemble model is better than the base model in accuracy, and the F1-macro score of the ensemble model performs better with merged. Among them, the accuracy of the ensemble model is at least 2.6% higher than that of the base model, and at most 7%. The F1-macro score is improved by at least 2.1% and at most 6.8%. This fully illustrates that our ensemble learning model plays a very good role in improving the performance of the model. The more information the model inputs, the better the performance of the model, but too much data will affect the efficiency of model operation.
On the test data set, we only got two results: 1) not merging and removing stop words, 2) merging and not removing stop words. The F1-macro score of the first type is only 0.6975, and the F1-macro score of the second type reaches 0.7548 show in Table 2, which once again shows that data processing and ensemble learning methods can effectively improve the performance of the model.
This article describes the fake news detection classification task in the IberLEF 2021 task. We used some classic feature extraction methods and machine learning techniques, and achieved high performance on the development set through ensemble methods. Compared with other deep learning and machine learning methods, the performance on the test set is also very competitive. Compared with MEX-A3T 2020 [13], the accuracy rate on the verification set has increased by about 8%, and the F1-macro score has increased by 6%. Compared with last year's best papers, the results of our model are also very competitive. The best F1-macro score we got on the test set was 0.7548, which was the second place. Due to changes in language and tweet content, the performance on the development set is still lower The future work is to explore more advanced technologies, use better feature extraction methods, and achieve better results in the next competition. Secondly, we also plan to apply our model to other languages and better solve the current flood of Covid-19 information. Finally, it is harmful to treat all news from a link as fake news or real news. We will solve this problem in future work.
We would like to thank the organizers for the opportunity and organization of this task, as well as teachers and seniors for their help. Finally, we would like to thank the school for supporting my research and the patient work of future reviewers.