HAHA at FakeDeS 2021: A Fake News Detection Method Based on TF-IDF and Ensemble Machine Learning Kun Li School of Information Science and Engineering, Yunnan University, Yunnan, P.R. China 2106967047@qq.com Abstract. This paper describes our participation in the FakeDeS [5] Task at Iberlef 2021: Fake News Detection in Spanish. Base on this task, we propose the classic TF-IDF feature extraction technology and Stack- ing ensemble learning method base on weak classifiers. It not only an- alyzes the content of the news, but also combines effective information such as publishers and topics to improve the performance of our model. We used five machine learning models, and achieved very competitive results on both the validation set and the test set, and got the second place in the final evaluation phase. Keywords: Fake News Detection, TF-IDF, Machine Learning, Ensem- ble Model 1 Introduction Fake news refers to a kind of public opinion that uses false information to de- ceive the parties in order to achieve a certain purpose. It fails to truly reflect the original appearance of objective things and contains false elements. The in- formation provided by fake news is designed to manipulate people for different purposes: terrorism, political election, advertising, satire, etc. In social networks, fake news spreads in seconds among thousands of people and research has shown that misinformation spreads faster, farther, deeper, and more widely than true information [12], so it is necessary to develop tools to help control the amount of fake news on the network. A few years ago, the method of detecting fake news was mainly to analyze the effective features from various sources, including the content of the text, user data and the form of news dissemination. It mainly distinguishes true and false news from the aspect of language features, such as writing style and special headlines [7], vocabulary and syntactic analysis [10]. In addition to language features [3], some studies have proposed classification schemes on user features IberLEF 2021, September 2021, Málaga, Spain. Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). and time features [7]. Recent fake news detection methods mainly use machine learning and deep learning techniques, with special attention to language-based methods [8, 11, 14, 15]. Some people use TF-IDF feature extraction technology for fake news detection and have achieved good results [1, 13, 2]. A fake news detection system is designed to help user detect and filter po- tentially deceptive news. A predictive method of deliberately misleading news is based on the analysis of the real and deceptive previously censored news, that is, the annotated corpus. Text is main carrier of news information, and the study of news text helps to effectively identify fake news. The specific task is: given the text of a news event, determine whether the event is real news of fake news. For the evaluation of systems, we will use a new testing corpus containing news related to COVID-19 and news from other Ibero-American countries. Its avail- ability will introduce two main challenges to the task: thematic and language variation. Our systems need to take into consideration that part of the testing corpus contains news in a topic that does not exist in training corpus, likewise, we should take into account that the other part of the testing corpus contains news in a different variation of the Spanish that is in training corpus. This paper proposes a method for fake news detection: A fake news detection method based on TF-IDF and ensemble machine learning. TF-IDF has the characteristics of simplicity, fast calculation. and it performs well for processing long texts. The Section 2 introduces the corpus and analyzes the composition and distribution of the data. The third section introduce the methodology, data processing methods, feature extraction methods, the base model used, and the final ensemble model. The experiments sett and results are presented in Section 4. Finally, Section 5 outlines the final conclusions and future work. 2 Corpus Description The Spanish fake news corpus [9] is news collected from several online sources: existing newspaper websites, media company websites, special websites dedicated to verifying fake news, and websites designated by different reporters as regular fake news publications. All these articles are written in Mexican Spanish. The corpus collected 971 news items from different sources from January to July 2018: 971 news items were divided into training set and test set. Among them, there are 676 pieces of data in training set and 295 pieces of data in the test set. Only two categories (True of Fake) are considered for the marking of the corpus, and the specific conditions of each piece of data are as follows: – Category: Fake/ True. – Topic: Science/ Sport/ Economy/ Education/ Entertainment/ Politics, Health/ Security/ Society. – Headline: The title of the news. – Text: The complete text of the news. – Link: The URL where the news was published. Among them, the number of fake news and real news is fairly balanced. And the number of fake news and real news in the Topic column is almost the same. However, there is a big gap between the authenticity of news published on differ- ent websites. Some websites are almost all fake news, and there are also websites that are all real news. This provides a good idea for our feature extraction. We will consider the impact of “Link” and “Topic” on the classification results when we do experiments. 3 Method and Technology This section includes 4 parts: data preprocessing, feature extraction methods, classification models, and ensemble model methods. 3.1 Data Preprocessing In order to get better results, data preprocessing is essential. And data pre- processing is usually the first step in natural language processing tasks. First intercept the most critical information in Link, and the result after interception looks like this: www.elruinaversal.com. Because we have analyzed the sources of fake news and found that almost all the news on some websites is fake news. Then we observe the data and find that each row consists of Category, Topic, Source, Headline, Text and Link. All the contents in Category, Topic, Source, Headline, Text and Link are merged as our new input. Finally, data cleaning is performed on the merged input data. Perform data cleaning on the merged data, use regular expressions to remove links, special symbols, punctuation, etc. According to the length of the text, the length of the longest text is 2578 and the shortest text is 31. So we decided to use nltk to remove the stop words in the text. The longest text length after removing the stop words is 1379, and the shortest is 18. Removing stop words in the text will reduce the effect of the model, and not removing stop words will improve the performance of the model. However, we have proved through experiments that removing stop words will reduce the performance of the model, which will be explained in subsequent experiments. And we will verify it in the next experiment. The data processing is: 1) merge all columns + remove stop words: 2) merge all columns + without remove stop words: 3) only Text + remove stop words: 4) only Text + without remove stop words. 3.2 Feature Extraction The method base on news content focuses on extracting various features of fake news content, including knowledge-base and style-based features. This paper mainly uses two methods to extract text features: 1) LabelEncoder; 2) TF-IDF. – LabelEncoder: We use Sklearn’s LabelEncoder method to hard-code text features, that is, to encode discrete numbers or text, and convert the dis- crete data to numbers between (0, n-1), where n represents different data values. We performed LabelEncoder on the Topic and Source features. The LabelEncoder method also played a very good role in the experiment. – TF-IDF: Term Frequency-inverse Document Frequency (TF-IDF) is a sta- tistical analysis method for keywords, used to evaluate the importance of a word to a document set or a corpus. The importance of a word is positively correlated with the number of times it appears in the article, and negatively correlated with the number of times it appears in the corpus. TF-IDF can effectively avoid the influence of commonly used words on keywords and im- prove the relevance between keywords and articles. TF refers to the total number of times a word appears in the article. This indicator is usually nor- malized and defined as the number of times a word appears in the article divided by the total number of words in the article, which can prevent the result from being biased towards too long document (The same word usually has a higher word frequency in a long document than in a short document). IDF refers to the frequency of reverse documents. The fewer documents that contain a word, the greater of the IDF value, indicating that the word has a strong ability to distinguish. Using TF-IDF can well extract text features in Spanish news. For texts with a length of several thousand, TF-IDF is better than RNN and other neural network model in extracting features of long texts. And for the challenge of changing language, TF-IDF can easily solve it. 3.3 Base Classification Model We use five basic weak classifiers as our base model. – LogisticRegression (LR): Logistic regression is used to discover the connec- tion between features and output results, used for classification problems in supervised machine learning algorithms, and has a close relationship with neural networks. Neural networks can be regarded as multiple logistic re- gression classifiers Stacked. Logistic regression can be used for binary clas- sification problems and multi-classification problems. – SGDClassifier (SGDC): Mainly used in large-scale sparse data problems. It is a collection of linear classifiers trained with stochastic gradient descent algorithm, It is a linear (soft interval) support vector machine classifier by default, which is logistic regression in this article. – PassiveAggressiveClassifier (PAC): It is a classic online linear classifier, which can continuously integrate new samples to adjust the classification model and enhance the classification ability of the model. It can perform feature extraction on streaming data, and can perform incremental learning. – RidgeClassifier (RC): This classifier uses penalized least square function to adapt to the classification model. The loss function used by RidgeClassifier can make different calculation performance profiles. – LnearSVC (LSVC): A linear classification supports vector machine is imple- mented by liblinear, which can be used for two-class classification or multi- class classification. 3.4 Ensemble Model LightGBM [6]: The model is a fast, distributed, high-performance gradient boost- ing framework base on decision tree algorithm. It can be used in sorting, clas- sification, regression, and many other machine learning tasks. LightGBM is an improvement of the GBDT algorithm [4]. LightGBM didn’t use the traditional pre-sorting idea, but optimizes the histogram of the eigenvalues. The weak clas- sifier is used to iteratively train to obtain the optimal model, which has the advantages of good training effect and not easy to overfit. The train method is GBDT: it is an algorithm that classifies or regresses data by using an additive model and continuously reducing the residuals generated during the training process. Fig. 1. Combine the processed data of the five basic models and then perform ensemble learning. Choose the above LogisticRegression (LR), SGDClassifier (SGDC), PassiveAg- gressiveClassifier (PAC), RidgeClassifier (RC), LnearSVC (LSVC) as the weak base model. Figure 1 shows the use of the Stacking method in the ensemble learning method to predict all the trained base models on the entire training set, and each base model will get a classification prediction result. For each base model, we train out model by using 5-fold cross-validation, concatenate the clas- sification prediction results after each base model training, and finally send all the features to the final LightGBM model for training. 4 Experiments 4.1 Experimental Setup First, data processing is performed, and then all the hyperparameter of the ex- periment are introduced. Almost all base model use default parameters, and each base model uses 5-fold cross-validation for training; the LightGBM model also uses 5-fold cross-validation for training. The training model of the LightGBM model is GBDT; the learning rate is set to 0.01; the maximum number of itera- tions num-boost-round is set to 10000; the progress is displayed every 50 itera- tions. Finally, the model outputs the final accuracy rate and and F1-macro. The hyperparameters of each classifier are as follows: LR (random-state=1017, C=3), SGDC (random-state=1017, loss=‘log’), PAC (random-state=1017, C=2), RC (random-state=1017), LSVC (random-state=1017). 4.2 Result We will evaluate the model from F1 measure and accuracy on the “fake” class. The results of the base model and ensemble model on the training data shown in Table 1, where “Merge-All” is to concatenate all the information, and “Stop- words” is removing stop words from the text. Accuracy and F1-macro are both results obtained on the validation set. From Table 1, we can see that the per- Table 1. Results for the development data set of fake news by using different classifier models, and preprocessing techniques. Model Merge All Stopwords Accuracy F1-macro LR Yes Yes 0.876 0.880 SGDC Yes Yes 0.883 0.890 PAC Yes Yes 0.885 0.892 RC Yes Yes 0.873 0.878 LS Yes Yes 0.879 0.883 Ensemble Yes Yes 0.911 0.913 LR Yes No 0.859 0.861 SGDC Yes No 0.865 0.871 PAC Yes No 0.870 0.875 RC Yes No 0.862 0.865 LS Yes No 0.865 0.869 Ensemble Yes No 0.902 0.904 LR No Yes 0.808 0.805 SGDC No Yes 0.811 0.816 PAC No Yes 0.805 0.813 RC No Yes 0.817 0.815 LS No Yes 0.812 0.812 Ensemble No Yes 0.882 0.884 LR No No 0.822 0.819 SGDC No No 0.822 0.828 PAC No No 0.827 0.832 RC No No 0.819 0.816 LS No No 0.828 0.828 Ensemble No No 0.898 0.900 formance of the ensemble model is always better than that of the base model, regardless of whether the merging and stop words are removed. For the same model, the highest accuracy can be obtained by performing two data processing methods at the same time. 91.1% and the highest F1-macro score 91.3%. At the same time, removing the stop words actually weakens the performance of the model, and merging all column information can significantly improve the performance of the base model, and it also improves the ensemble model to a certain extent. Table 2. The results of our model on the official test sets. Model F1-macro rank Ensemble 0.7548 2 We can get that in the same base model, merge can play a certain role in improving the accuracy of the model and the F1-macro score. Without the merge, the accuracy of the model is significantly reduced, and the remove stop words from the text will reduce the performance of the model. At the same time, no matter how the data is processed, the ensemble model is better than the base model in accuracy, and the F1-macro score of the ensemble model performs better with merged. Among them, the accuracy of the ensemble model is at least 2.6% higher than that of the base model, and at most 7%. The F1-macro score is improved by at least 2.1% and at most 6.8%. This fully illustrates that our ensemble learning model plays a very good role in improving the performance of the model. The more information the model inputs, the better the performance of the model, but too much data will affect the efficiency of model operation. On the test data set, we only got two results: 1) not merging and removing stop words, 2) merging and not removing stop words. The F1-macro score of the first type is only 0.6975, and the F1-macro score of the second type reaches 0.7548 show in Table 2, which once again shows that data processing and ensemble learning methods can effectively improve the performance of the model. 5 Conclusions and Further Work This article describes the fake news detection classification task in the IberLEF 2021 task. We used some classic feature extraction methods and machine learn- ing techniques, and achieved high performance on the development set through ensemble methods. Compared with other deep learning and machine learning methods, the performance on the test set is also very competitive. Compared with MEX-A3T 2020 [13], the accuracy rate on the verification set has increased by about 8%, and the F1-macro score has increased by 6%. Compared with last year’s best papers, the results of our model are also very competitive. The best F1-macro score we got on the test set was 0.7548, which was the second place. Due to changes in language and tweet content, the performance on the development set is still lower The future work is to explore more advanced technologies, use better feature extraction methods, and achieve better results in the next competition. Secondly, we also plan to apply our model to other languages and better solve the current flood of Covid-19 information. Finally, it is harmful to treat all news from a link as fake news or real news. We will solve this problem in future work. Acknowledgments We would like to thank the organizers for the opportunity and organization of this task, as well as teachers and seniors for their help. Finally, we would like to thank the school for supporting my research and the patient work of future reviewers. References 1. Ahmed, H., Traore, I., Saad, S.: Detecting opinion spams and fake news using text classification. Security and Privacy 1(1), e9 (2018). https://doi.org/https://doi.org/10.1002/spy2.9, https://onlinelibrary.wiley.com/doi/abs/10.1002/spy2.9 2. Arce-Cardenas, S., Fajardo-Delgado, D., Carmona, M.Á.Á.: Tecnm at MEX- A3T 2020: Fake news and aggressiveness analysis in mexican spanish. In: Iber- LEF@SEPLN. CEUR Workshop Proceedings, vol. 2664, pp. 265–272. CEUR- WS.org (2020) 3. Castillo, C., Mendoza, M., Poblete, B.: Information credibility on twit- ter. In: Proceedings of the 20th International Conference on World Wide Web. p. 675–684. WWW ’11, Association for Computing Machin- ery, New York, NY, USA (2011). https://doi.org/10.1145/1963405.1963500, https://doi.org/10.1145/1963405.1963500 4. Friedman, J.H.: Greedy function approximation: A gra- dient boostingmachine. The Annals of Statistics 29(5), 1189 – 1232 (2001). https://doi.org/10.1214/aos/1013203451, https://doi.org/10.1214/aos/1013203451 5. Gómez-Adorno, H., Posadas-Durán, J.P., Bel-Enguix, G., Porto, C.: Overview of fakedes task at iberlef 2020: Fake news detection in spanish. Procesamiento del Lenguaje Natural 67(0) (2021) 6. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.Y.: Lightgbm: A highly efficient gradient boosting decision tree. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 3149–3157. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017) 7. Kwon, S., Cha, M., Jung, K., Chen, W., Wang, Y.: Prominent features of rumor propagation in online social media. In: 2013 IEEE 13th International Conference on Data Mining. pp. 1103–1108 (2013). https://doi.org/10.1109/ICDM.2013.61 8. Oshikawa, R., Qian, J., Wang, W.Y.: A survey on natural language processing for fake news detection. In: Proceedings of the 12th Language Resources and Evalua- tion Conference. pp. 6086–6093. European Language Resources Association, Mar- seille, France (May 2020), https://www.aclweb.org/anthology/2020.lrec-1.747 9. Posadas-Durán, J.P., Gómez-Adorno, H., Sidorov, G., Escobar, J.J.M.: Detection of fake news in a new corpus for the spanish language. Journal of Intelligent & Fuzzy Systems 36(5), 4869–4876 (2019) 10. Potthast, M., Kiesel, J., Reinartz, K., Bevendorff, J., Stein, B.: A stylo- metric inquiry into hyperpartisan and fake news. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pp. 231–240. Association for Computational Linguis- tics, Melbourne, Australia (Jul 2018). https://doi.org/10.18653/v1/P18-1022, https://www.aclweb.org/anthology/P18-1022 11. Shu, K., Sliva, A., Wang, S., Tang, J., Liu, H.: Fake news detec- tion on social media: A data mining perspective. SIGKDD Explor. Newsl. 19(1), 22–36 (Sep 2017). https://doi.org/10.1145/3137597.3137600, https://doi.org/10.1145/3137597.3137600 12. Vosoughi, S., Roy, D., Aral, S.: The spread of true and false news online. Science 359(6380), 1146–1151 (2018). https://doi.org/10.1126/science.aap9559, https://science.sciencemag.org/content/359/6380/1146 13. Zaizar-Gutiérrez, D., Fajardo-Delgado, D., Carmona, M.Á.Á.: Itcg’s participation at MEX-A3T 2020: Aggressive identification and fake news detection based on tex- tual features for mexican spanish. In: Cumbreras, M.Á.G., Gonzalo, J., Cámara, E.M., Martı́nez-Unanue, R., Rosso, P., Zafra, S.M.J., Zambrano, J.A.O., Miranda, A., Zamorano, J.P., Gutiérrez, Y., Rosá, A., Montes-y-Gómez, M., Vega, M.G. (eds.) Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020) co- located with 36th Conference of the Spanish Society for Natural Language Process- ing (SEPLN 2020), Málaga, Spain, September 23th, 2020. CEUR Workshop Pro- ceedings, vol. 2664, pp. 258–264. CEUR-WS.org (2020), http://ceur-ws.org/Vol- 2664/mexa3t paper4.pdf 14. Zhang, X., Ghorbani, A.A.: An overview of online fake news: Charac- terization, detection, and discussion. Information Processing Management 57(2), 102025 (2020). https://doi.org/https://doi.org/10.1016/j.ipm.2019.03.004, https://www.sciencedirect.com/science/article/pii/S0306457318306794 15. Zhou, X., Zafarani, R., Shu, K., Liu, H.: Fake news: Fundamental theories, detec- tion strategies and challenges. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. p. 836–837. WSDM ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/836–837, https://doi.org/10.1145/3289600.3291382