In this work, an automated system of document analysis and classification based on deep learning methods has been developed. The system is able to process text data using both traditional models (BoW+LR and TF-IDF+LR) and deep learning models (LSTM, BERT). The system allows users to select a data set, perform text preprocessing, perform classification using four approaches, and receive detailed reports on the performance of the models. The main attention is paid to ocmparing the effectiveness of the methods, which is implemented through the evaluation of metrics (accuracy, completeness, F1-measure) and visualization of the results. The BERT model showed the highest accuracy. The testing confirmed the system's functionality.
The modern world is characterized by a constant increase in the amount of information, which requires effective methods of processing, analyzing and structuring it. This problem is especially acute in the context of text documents, the number of which is growing exponentially due to the development of the Internet, social networks, electronic libraries and digital archives.
In modern information systems, document analysis and classification are fundamental processes
for effective management, processing and interpretation of information. Thanks to these processes,
organizations are able not only to store large volumes of data, but also toysstematize them, highlight
key patterns and use the results obtained to make informed decisions in business, science, education
and other industries [
The relevance of the work lies in the fact that the development of a document analysis and classification system based on deep learning allows to significantly increase the efficiency of processing large volumes of text data, ensuring high accuracy and automation of processes.
The purpose of the research is to develop an automated document analysis and classification system
using deep learning methods to optimize text data processing processes, improve classification accuracy, and automate information analysis.
The rapid development of machine learning allows simplifying research in various fields of science. The application of machine learning methods for document classification is a relevant direction, using achievements to solve practical problems.
The article [
The study [
The article [
The problem of organizing software and algorithms for neural networks for natural language
processing is considered in the work [
The study [
The article [7] presents an approach to building a model of an intelligent document management system using machine learning methods to ensure effective work of employees in organizations. A number of problems were solved to optimize each of the document management subsystems, as a result of which a model of an intelligent document management system was developed, which can be effectively applied in enterprises, state and corporate institutions. At the same time, the paper considers the application of thematic modeling methods and text analysis algorithms based on a multi-agent approach, which can be used to build an intelligent document management system.
The multimodal deep learning architecture TechDoc is presented in [8]. The model uses three types of information, including natural language texts and descriptive images in documents, as well as connections between documents. The architecture synthesizes a convolutional neural network, a recurrent neural network and a graph neural network using an integrated learning process. The architecture was applied to a large multimodal database of technical documents, and the model was trained to classify documents based on the hierarchical system of the International Patent Classification. The results show that TechDoc has higher classification accuracy than unimodal methods and other modern benchmarks.
In the work [9] an analysis and comparison of recurrent neural networks (RNN), in particular the LSTM and GRU models, was carried out to determine the effectiveness of these models in analyzing the sentiment of text messages. A public dataset from the Twitter social networking platform was used to train the model. The obtained research results allow us to conclude that compared to LSTM, the GRU model was more effective in the task of text classification.
The developed document analysis and classification system using is a software package designed for automatic processing of texts from various domains (news, movie reviews, medical transcriptions) and determining their category or sentiment. The main approach to classification is to use both traditional machine learning methods Bag of Words and TF-IDF with logistic regression (BoW+LR and TF-IDF+LR), and deep learning models, in particular the recurrent neural network LSTM and the BERT (Bidirectional Encoder Representations from Transformers) architecture. The system provides flexibility in choosing a data set, comparing the effectiveness of different methods, and creating visualized reports with the results.
Modern document analysis and classification systems use a variety of approaches, including traditional machine learning methods and deep learning technologies. Traditional methods, such as BoW+LR and TF-IDF+LR, are easy to implement and effective for small or structured data sets, but they have limitations in detecting complex semantic relationships and processing large, diverse text data. In contrast, deep learning methods such as recurrent neural networks (RNNs, including LSTM) and transformers (e.g., BERT) provide significantly higher accuracy and flexibility when dealing with texts containing nuance, context, and noise.
Traditional methods (BoW+LR and TF-IDF+LR) are fast and suitable for small data sets, but they do not take into account semantic relationships and context, which limits their effectiveness when working with texts with complex structure or ambiguity (for example, medical terms or sarcastic reviews). In contrast, LSTM and BERT allow modeling long text dependencies and semantic relationships, which is crucial for accurate classification of documents in areas such as news, reviews or medical transcriptions.
It should be noted that deep learning methods face problems, especially in terms of interpreting the obtained results.
Taking into account these limitations, system architecture is presented, designed according to the modular principle, which allows it to be easily adapted to new data sets or models (Fig. 1).
With Figure 1, it can be seen that the system has a modular architecture, consisting of separate components that are responsible for: 1. Data entry (selection and loading of the data set); 2. Text preprocessing; 3. Document classification using traditional methods (BoW+LR and TF-IDF+LR) and deep learning models (LSTM, BERT); 4. Data analysis (visualization of category distribution); 5. Evaluation of results and comparison of models; 6. Storage of trained models.
This architecture ensures its scalability and extensibility. Experimental research into the effectiveness of the proposed document classification model is aimed at developing a methodology for evaluating four approaches: BoW+LR and TF-IDF+LR, LSTM and BERT [10, 11].
The research methodology is based on a literature review and theoretical modeling of the document classification process using deep learning methods and classical approaches. Let us list the main stages of the research.
7. Data selection - to assess the adaptability of the models to different types of documents, it is
planned to use various text corpora, such as news articles, user reviews and specialized texts
(for example, news sets) [
The methodology takes into account the influence of factors such as the volume of data, its structure and stylistic features, which may affect the classification results [17]. The developed methodology and experimental design create the basis for evaluating BoW+LR, TF-IDF+LR, LSTM and BERT in document classification. The selected metrics (accuracy, completeness, F1-measure) will provide a comprehensive analysis of the effectiveness [18]. The main aspects are given in Table 1. Theoretical expectations indicate an accuracy advantage for BERT, but practical testing will refine these assumptions [10].
Data preparation for deep learning models is a critical step in the document analysis and classification system, which ensures high-quality processing of text data from three datasets - News Category Dataset, IMDB Movie Reviews and medical transcriptions - for further use in LSTM and BERT models in the document classification module. Data tokenization, vectorization and optimization were performed.
To demonstrate the results, the News Category Dataset [19] dataset was selected, which contains the fields: category, title, authors, links, short description and news publication date. The performance of four models BoW+LR, TF-IDF+LR, LSTM and BERT is compared on this corpus.
The goal was not only to determine the best approach, but also to identify the strengths and weaknesses of each model, as well as to develop recommendations for improving the system, taking into account its practical application in real conditions. Testing included several stages: system startup, data structure analysis, performance metrics evaluation (accuracy, completeness, F1measure), results visualization and model performance comparison for different usage scenarios.
Testing began with system initialization, which allowed us to check its basic performance and preparation for data processing. The output data (Figure 2) is presented, which reflect the system startup and initial configuration.
The console shows the successful loading of stopwords from the /kaggle/input/nltk-stopwords directory, GPU availability (True), which positively affects the processing speed of deep models, the TensorFlow version (2.17.1), and the dataset selection menu. The user selected the News Category Dataset (1).
Before classification, an initial analysis of the structure of the News Category Dataset dataset was performed to assess its complexity and identify key categories for testing. A graph (Figure 3) is shown, which displays the distribution of categories in the dataset. The first 15 categories are displayed.
The histogram displays the frequencies of news categories in descending order. POLITICS has the highest values, followed by WELLNESS and ENTERTAINMENT, while other categories are characterized by significantly lower frequencies. This justifies the selection of the ten most common categories for further processing. The analysis revealed an uneven distribution, which affected the classification strategies.
The performance testing of the models was carried out on a dataset of the top 10 categories of the News Category Dataset. Each model was evaluated by the metrics of accuracy, completeness and F1measure (macro/weighted), the results are presented in the form of tables, confusion matrices and graphs. This approach allowed us to analyze in detail the effectiveness of each approach and identify their features. Figure 4 shows the results of the BoW model on a given sample.
The Figure 4 displays the number of records for each category, which is used for pre-filtering the data before classification. Figures 5 and 6 contain the classification report of the BoW+LR and TF IDF+LR models. Note that they were trained with class balancing, and the optimal hyperparameters were determined using the GridSearchCV method with five-fold cross-validation.
Within the study, two models - BoW+LR and TF-IDF+LR were compared. Both approaches showed similar results (accuracy 0.765; macro-F1 0.72; weighted-F1 0.77), with TF-IDF+LR providing a small but stable improvement in weighted F1 compared to BoW+LR.
Confusion matrix analysis shows misclassification between thematically adjacent classes. The transition from BoW+LR to TF-IDF+LR moderately increases the proportion of correct classifications and reduces cross-over errors. In the course of the study, an LSTM neural network was built, the architecture of which is presented in Figure 9.
The figure shows that the model contains two recurrent layers, as well as additional levels for representation formation, regularization, feature aggregation and classification.
AdamW was used as the optimizer, and categorical crossentropy was used as the loss function. During training, early stopping and speed reduction were applied to prevent overtraining; as a result, the model was trained for 5 epochs. The graphs (Figure 10) are shown, which show the dynamics of LSTM learning. The graphs illustrate the reduction in loss and increase in accuracy during training.
A Figure 11 is provided, which contains the classification report for the LSTM model.
A confusion matrix (Figure 12) is given, which shows the distribution of LSTM predictions.
From Figures 11 and 12, we can conclude that compared to the BoW+LR and TF-IDF+LR models, the LSTM demonstrates worse integral indicators: accuracy 0.713 vs. 0.765, macro-F1 0.611 vs. 0.72, weighted-F1 0.7 vs. 0.77. The confusion matrix for the LSTM has a larger off -diagonal dispersion, especially for less represented classes, local improvements are observed only for individual categories. Therefore, on short texts (title + short description) TF-IDF+LR is a stronger baseline approach, while LSTM requires more powerful features or contextual models to achieve an advantage.
In this study, the BERT model was used for the text classification task. The input sequences were formed by the WordPiece tokenizer, and the token indices and attention mask were passed to the model. Training was performed using fine-tuning over 6 epochs with a batch size of 16 (to limit memory consumption). Optimization was performed using AdamW with a linear learning rate decay. Categorical crossentropy was used as the loss function. The model architecture is shown in Figure 13.
The learning curves (Figure 14) are shown, which demonstrate the BERT learning process. The graphs show that the accuracy increases during the training process, the loss function approaches zero.
The next Figure 15 is provided, which contains the classification report of the BERT model, which shows an overall accuracy of 0.793, with a macro-F1 of 0.740 and a weighted-F1 of 0.786.
A confusion matrix (Figure 16) is presented, which illustrates BERT predictions. The matrix shows a predominance of diagonal elements, which indicates a higher accuracy of BERT compared to the baseline models. Errors are single and mainly fall on semantically close classes, which confirm the better recognition ability of the contextual model on short texts.
The final stage of testing involved comparing all models to determine their efficiency and practical applicability. A histogram (Figure 17) is presented, which reflects the comparison of the accuracy of all system models (BoW+LR, TF-IDF+LR, LSTM and BERT) for the News Category Dataset, created by the results evaluation and model comparison module.
The histogram shows that the BERT model yields the best result, with a score of 0.793. The BoW+LR and TF-IDF+LR models give similar values (0.765-0.766), while LSTM is significantly inferior (0.713). Testing confirmed the system's performance and its ability to classify various texts with varying accuracy depending on the model. The recommendations aim to enhance the accuracy, speed, and scalability of the system, making it promising for practical use in document analysis automation.
This study covers the following stages: software architecture design, classification algorithm development, and system testing on real data. The developed system is capable of processing text data using both traditional methods (BoW+LR and TF-IDF+LR) and deep learning models (LSTM, BERT).
The system allows users to select a data set, perform text preprocessing, perform classification using four approaches, and receive detailed reports on model performance. The main focus is on comparing the effectiveness of methods, which is implemented through the evaluation of metrics (accuracy, completeness, F1-measure) and the visualization of results.
Key achievements of the developed system:
Classification with a maximum accuracy of 0.793 (BERT) has been implemented; Comparison of models with visualization of results is provided; Data for deep learning models (LSTM, BERT) is prepared;
Testing was carried out, which confirmed the system's operability.
Thus, the created system is a practical tool for automating document classification, which can be the basis for the further development of information systems in various industries. Its implementation is able to optimize the processing of large volumes of text. Development prospects include the integration of a web interface and improving models by optimizing them and using larger computing resources.
[7] M. Sambetbayeva, et al., Development of intelligent electronic document management system model based on machine learning methods, Eastern-European Journal of Enterprise Technologies 1(2) (2022) 68–76. doi:10.15587/1729-4061.2022.251689. [8] S. Jiang, J. Hu, C.L. Magee, J. Luo. "Deep learning for technical document classification". IEEE
Transactions on Engineering Management 71 (2024): 1163–1179. doi:10.1109/TEM.2022.3152216. [9] T. Filimonova, O. Pursky, V. Babenko, A. Nechepourenko, V. Shvets, V. Gamaliy, Text sentiment analysis using different types of recurrent neural networks. In: Proceedings of the 5th International Conference on Image Processing and Capsule Networks, ICIPCN, Dhulikhel, Nepal, 2024, pp. 383–387. doi:10.1109/ICIPCN63822.2024.00068. [10] Faizur Rashid, Suleiman M. A. Gargaare, Abdulkadir H. Aden, Afendi Abdi, Machine learning algorithms for document classification: Comparative analysis. International Journal of Advanced Computer Science and Applications (IJACSA) 13(4) (2022) 260-265. doi.org/10.14569/IJACSA.2022.0130430. [11] S. Leviner, Types of document classification methods. Charactell, 2023. URL: https://www.charactell.com/resources/types-of-document-classification-methods/. [12] D. Mwiti, Python bag of words model: A complete guide. DataCamp, 2024. URL: https://www.datacamp.com/tutorial/python-bag-of-words-model. [13] TfidfVectorizer. Scikit-learn Documentation, 2025. URL: https://scikitlearn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html. [14] LSTM layer. Keras Documentation, 2025. URL: https://keras.io/api/layers/recurrent_layers/lstm/. [15] BERT. Hugging Face Transformers Documentation, 2025. URL: https://huggingface.co/docs/transformers/model_doc/bert. [16] Y. HaCohen-Kerner, D. Miller, Y. Yigal, The influence of preprocessing on text classification using a bag-of-words representation, PLoS ONE, 15(5) 2020 e0232525. doi:10.1371/journal.pone.0232525. [17] L. Stewart, Document analysis – How to analyze text data for research. ATLAS.ti Research Hub, 2025. URL: https://atlasti.com/research-hub/document-analysis. [18] P. Kashyap, Understanding precision, recall, and F1 score metrics. Medium, 2024. URL: https://medium.com/@piyushkashyap045/understanding-precision-recall-and-f1-scoremetrics-ea219b908093. [19] R. Misra, R. News Category Dataset (Version 3) [Data set]. K aggle 2022. URL: https://www.kaggle.com/datasets/rmisra/news-category-dataset.