The increasing use of artificial neural networks for knowledge processing often lacks precise knowledge representation. To address this issue, we propose using a word-type-weighted Word2Vec model to achieve more accurate representations of individual words within sentences. Our approach incorporates weighting vector embeddings of words based on parts-ofspeech predictions generated by the spaCy library. Experimental results demonstrate that, compared to simple Word2Vec, our model enhances the accuracy of recognizing the semantics of a sentence, while maintaining significantly lower computational requirements than large language models and various variants of Transformer.
Recently, artificial intelligence (AI) and machine learning
(ML) have proved to be extremely useful in most scientific
ifelds [
The following section explains the concept of sentence embeddings and its applicability. Section 3 describes the
This section outlines the methodology used to develop the word-type weighted Word2Vec model, used to predict semantic similarity of sentences. Our approach integrates word embeddings with parts-of-speech information to improve accuracy without large processing costs.
The corpus we used was the Microsoft Research paraphrase corpus[8]. It contains around 5800 pairs of sentences. We trained the algorithm on the train set of this corpus and tested it on its test set.
We used the public
GoogleNews-vectorsnegative300[9] Word2Vec implementation, for more objective and clear results, because of how widespread this corpus is.
The model utilizes 300-dimensional vectors and has been trained with around 3 million diferent English words. The size of this model is around 1,6 GB.
Speech) tagging, because of its eficiency and precision, which is crucial to fine-tuning the weights correctly.
As the first step of preprocessing, we use the spaCy library to tag each word in a sentence, which as a result also tokenizes the given sentence. SpaCy assigns tags automatically, using a neural network. Then we delete all the symbols. After deleting the symbols, we apply a standard spell-checking algorithm to correct the mistakes created by deleting the symbols. After that we employ our embedding algorithm.
This embedding algorithm starts by verifying that the word is not a stop word. If it passes this check, we clarify whether the word is present in our model. If the word is absent, we proceed to lemmatization and check again, followed by stemming and another check. If all of these steps are unsuccessful, we assign to each token the embedding based on its assigned tag. For instance, the embedding of John is assigned to every first name tagged as a proper noun because there are missing embeddings for them.
In this study, we consider weights for each word type, denoted as wt, where wt is the index of the word type.
For each wt, we assume that wt ∈ Q.
We used the spaCy library [10] for POS (Parts-of- the Hessian matrix (or its inverse) at each iteration to
We first divided the training text into two parts. First being 60 percent of the text and second being 40 percent of the text. We then used our text preprocessor to vector these parts of text. Both parts were made up of pairs of sentences, where half of them had the same semantic similarity and half did not.
Initially, we needed to make a suficiently accurate guess close to the global minimum. To achieve this, we used the Broyden-Fletcher-Goldfarb-Shanno algorithm, also known as the BFGS method, to minimize the mean squared error [11]. We opted for this method because, when tested, it was shown to be the most accurate for this specific type of problem.
The BFGS algorithm is an iterative method for solving unconstrained nonlinear optimization problems. It belongs to the family of quasi-Newton methods, which are used to find local maxima or minima of functions. The key idea behind BFGS is to update an approximation to improve the convergence rate. trix +1 is given by:
The BFGS update formula for the inverse Hessian ma+1 = th 1 g i e W 0 −1 −2
J D A
P D A
V D A
X U A
J N O C C
T E D
J T N I POS tag
N U O N
M U N
T R A P
N O R P
N P O R P
J N O C S
B R E V
The BFGS method was quite dependent on the initial conditions, and hence we did a number of iterations of this function while changing the text that was supposed to be similar or not. Afterward, we fitted a Gaussian distribution on the given ratio between weights because we considered the ratio more important than the finalized weights. Figure 1 depicts the weight ratios obtained in Table 1. We normalized the overall distribution around zero.
Subsequently, we generated random samples from the obtained Gaussian distribution. These samples were generally similar (Figure 1), although there were a few exceptions, such as with nouns, created from the larger ratio diferences.
Although some estimates were worse than others, all the diferences could be rectified, with the method we employed at last. We refined the weights, that were different, through an iterative process, comparing them with weights derived from the Gaussian distribution with small enough diferences. The refinement was achieved by minimizing logistic loss using the Nelder- Table 2 Mead method. The logistic loss was calculated based on Summary of Experimental Setup the prediction accuracy. Category
The logistic loss for a binary classification problem, Dataset also known as log-loss or binary cross-entropy loss, is Minimization Algorithms given by: Error Functions
Microsoft Research Paper BFGS, Nelder-Mead Logistic Loss, Mean Squared Error Gaussian Distribution Accuracy, F1-Score, AUC 10 60 % : 40 % 1 ∑︁ [ log (x · w) (w) = − =1
+(1 − ) log(1 − (x · w))] (2)
The Nelder-Mead algorithm minimizes the logistic loss function by iteratively refining a simplex with + 1 vertices in an -dimensional space [12]. The NelderMead method is particularly efective for optimizing the logistic loss function in logistic regression, especially in cases where the gradient is unavailable or the function is non-smooth. Through successive adjustments of the simplex vertices via reflection, expansion, contraction, and shrinkage, the algorithm steadily progresses toward the minimum of the logistic loss function.
Embeddings were too dependent on the length of their sentences. We have created a gradient-based weight creator, which modifies the embedding. It adds a corrector multiplied by the count of tokens in the sentence. We chose to use the additive weighted count of tokens in a sentence because, after many tests with diferent corrections, such as modification by the count of particular word types and multiplication with a weighted count of tokens, it was shown to be the most diferentiating factor between diferent sentences.
In table 2, you can see the full experimental setup of the methodology.
The resulting final weights were in some cases negative, with nouns being overly positive. Adjectives, nouns, 1benchmark results of available sentence transformers: https://www. sbert.net/docs/sentence_transformer/pretrained_models.html Assumed Distribution Evaluation Metrics Number of Executions
Training-Testing Set Ratio numerals, and verbs had the largest weights, while other parts of speech, for instance determiners or adpositions had weights close to zero. This most likely happened due to these POS having such large impact on sentences. The ifnal weights are shown in table 3. We have compared our approach to the BERT (Bidirectional Encoder Representations from Transformers) [13] fine-tuned for sentence embeddings, namely allMiniLM-L12-v2 which has good benchmark results1 and simple averaging Word2Vec without weighting. All results in this test have been obtained from the independent testing dataset. The testing dataset is balanced to contain the same number of records for each class (the same and diferent descriptions). We used the Accuracy, F1 score, and AUC[14] for measuring all the statistics.
This work was supported by the Grant Agency of the Czech Technical University in Prague, grant No. SGS23/205/OHK3/3T/18 and by the German Research Foundation (DFG) funded project 467401796. [7] M. Zhou, D. Liu, Y. Zheng, Q. Zhu, P. Guo, A text sentiment classification model using double word embedding methods, Multimedia Tools and Applications 81 (2022) 18993–19012. URL: https://doi.org/10.1007/s11042-020-09846-x. doi:10. 1007/s11042-020-09846-x. [8] W. B. Dolan, C. Brockett, Microsoft research paraphrase corpus, Microsoft Research, 2005. URL: https://www.microsoft.com/en-us/download/ details.aspx?id=52398, accessed: August 13, 2024. [9] T. Mikolov, K. Chen, G. Corrado, J. Dean, Efifcient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781 (2013). URL: https://github.com/mmihaltz/ word2vec-GoogleNews-vectors, accessed: August 13, 2024. [10] M. Honnibal, I. Montani, S. Van Landeghem, A. Boyd, spaCy: Industrial-strength Natural Language Processing in Python (2020). doi:10.5281/ zenodo.1212303. [11] C. T. Kelley, Iterative Methods for Optimization, SIAM, 1999, pp. 71–86. URL: https://epubs.siam. org/doi/abs/10.1137/1.9781611970920.ch4. doi:10. 1137/1.9781611970920.ch4. [12] J. A. Nelder, R. Mead, A simplex method for function minimization, The Computer Journal 7 (1965) 308– 313. URL: https://academic.oup.com/comjnl/article/ 7/4/308/354237. doi:10.1093/comjnl/7.4.308. [13] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URL: https://aclanthology.org/N19-1423. doi:10.18653/ v1/N19-1423. [14] C. Ferri, J. Hernández-Orallo, R. Modroiu, Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation, Springer, 2009. [15] A. Benavoli, G. Corani, F. Mangili, Should we really use post-hoc tests based on mean-ranks?, Journal of Machine Learning Research 17 (2016) 1–10. URL: http://jmlr.org/papers/v17/benavoli16a.html.