power. Knowledge Graph Embeddings (KGEs) are dense representations of entities and relations of a knowledge graph (KG) in a continuous vector space. In this paper, we present a preliminary analysis showing that KGEs can be substantially compressed by using only binary instead of continuous features, replacing costly floating point storage by a more space-eficient bitwise storage, while retaining the representational ∗Corresponding author.
CEUR ceur-ws.org
Knowledge Graph Embeddings (KGEs) are dense representations of entities and relations in
knowledge graphs, where each entity and relation is represented by a vector in a continuous
vector space. They are used for various in knowledge graph tasks, such as link prediction,
entity classification, for helping in knowledge-graph related tasks such as entity linking and
disambiguation, but also as knowledge representations in other downstream tasks, such as
recommender systems. [
While the scalability of KGE methods has been identified as a challenge [
To illustrate the issue of KGE size, we use the well-known knowledge graph DBpedia [
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). bit floating point values for representing embeddings by default. 2,3 Embedding DBpedia with 200 dimensions would thus yield a 6GB large KGE model.4
In this paper, we propose to use binary instead of continuous embeddings, i.e., embeddings that only use the values 0 and 1 for each dimension. By doing so, the size of embeddings can be reduced by a factor of 32 (using only 1 bit per entity and dimension instead of 32). In the example above, we could thus store a 200 dimensional binary embedding using less than 200MB.
The proposed approach is not limited to a single KGE method. Instead, we propose to binarize
an already trained KGE. In experiments with the DLCC node classification benchmark, eleven
diferent embedding methods, and diferent binarization methods, we show that the loss in
accuracy of using binary embeddings is rather small, but allows for a drastic reduction of storage
size. Especially for embeddings that are already computed for popular large-scale knowledge
graphs [
In this paper, we utilize two approaches for binarizing knowledge graph embeddings: a simple
dimension-based baseline, and an autoencoder-based approach adapted from [
Given that ()
is the embedding vector of an entity , we first compute the average vector as where () vector
is the i-th element of () from () as
. With that average vector, we can obtain a binary = () , all entities e () = { 0 () < 1 () ≥ In other words: every floating point value lower than the average for that dimension is represented by a 0, all others are represented by a 1.
Autoencoders are, in their simplest form, three-layer neural networks trained for dimensionality
reduction [
[
3https://pytorch.org/docs/stable/notes/numerical_accuracy.html 4200 dimensions × 7,620,000 entities × 32 Bit (1) (2) With that, the output of () is a binary vector. For further optimizing computations, they propose to align the dimensionality of the code layer to CPU register widths, such as 64, 128, and 256 bits. In this paper, we use their implementation5 for binarizing knowledge graph embeddings.
In order to evaluate the quality of binarized embeddings, we use the DBpedia portion of
DLCC [
Table 1 shows the average file sizes achieved with the diferent binarization methods. The
standard storage method uses an uncompressed TXT variant (i.e., storing the value ”0” or ”1” in
one byte), which can yield a compression factor up to 14. The .VEC format, which is supported
by the autoencoder implementation used in this paper [
DLCC has three sizes of problems for each task; here, we only present the results for largest problem size (5,000 examples per task). The full results are available online7; the results for the smaller groups (500 and 50 examples, resp.) are comparable to those on the largest task group.
Table 2 shows the results on the DLCC entity classification dataset. The average losses in accuracy are not very large, with the the baseline performing better than the smaller autoencoders. This makes us assume that the original embeddings capture the required information for the tasks at hand already at a very coarse level, i.e., it is suficient to know that a vector value for an entity at a particulary dimension is high or low, but it is not necessary to know how low it actually is.
Moreover, we observe the largest losses for ComplEx and DistMult, which are both tensor factorization methods, whereas RDF2vec SG has the smallest loss, compared to the uncompressed variant. Therefore, we conclude that binarization is not equally efective for each model, but that there are diferences by model family.
Furthermore, we can observe that the binarization has a smaller impact than the choice of the embedding variant: even the binarized versions of RDF2vec SG OA yield superior results to the original, i.e., non binarized embeddings of most other embedding methods.
In this paper, we have shown two strategies for reducing the size of a KGE model, one simple baseline using dimension-wise binarization, and an approach based on neural autoencoders. A promising preliminary evaluation on the DLCC entity classification benchmark shows that the size of embeddings can be drastically reduced at a comparatively low loss in downstream classification performance.
While the focus of this paper has been on optimizing the storage, we have not yet looked
into possible gains in computation time for downstream processing, i.e., at inference time. For
example, [
So far, we have only looked into tasks involving single entities, as defined in the gEval and DLCC benchmarks. The transfer to another popular usage of knowledge graph embeddings, i.e., link prediction and triple scoring, has not yet been investigated. Except for the B-CP approach [23], there are, to the best of our knowledge, no methods for exploiting binary knowledge graph embeddings for link prediction. In particular, since B-CP directly learns the embeddings, there are no studies on binarizing existing embeddings for link prediction.
In summary, the results have shown that binary knowledge graph embeddings can be an appealing alternative to the widely used floating point representations, allowing a drastic reduction of storage space while, at the same time, leading to only marginal loss in downstream performance. [cs]. [15] J. Portisch, H. Paulheim, Putting RDF2vec in Order, 2021. URL: http://arxiv.org/abs/2108.
05280. doi:10.48550/arXiv.2108.05280, arXiv:2108.05280 [cs]. [16] P. Ristoski, J. Rosati, T. Di Noia, R. De Leone, H. Paulheim, RDF2Vec: RDF graph embeddings and their applications, Semantic Web 10 (2019) 721–752. URL: https: //www.medra.org/servlet/aliasResolver?alias=iospress&doi=10.3233/SW-180317. doi:10. 3233/SW-180317. [17] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, O. Yakhnenko, Translating Embeddings for Modeling Multi-relational Data 2013 (2013). [18] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, G. Bouchard, Complex embeddings for simple link prediction, in: International conference on machine learning, PMLR, 2016, pp. 2071–2080. [19] B. Yang, W.-t. Yih, X. He, J. Gao, L. Deng, Embedding Entities and Relations for Learning and Inference in Knowledge Bases, 2014. URL: https://arxiv.org/abs/1412.6575v4. [20] M. Nickel, V. Tresp, P. Kröger, A Three-Way Model for Collective Learning on Multi
Relational Data., 2011, pp. 809–816. [21] Z. Sun, Z.-H. Deng, J.-Y. Nie, J. Tang, RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space, 2019. URL: http://arxiv.org/abs/1902.10197. doi:10.48550/ arXiv.1902.10197, arXiv:1902.10197 [cs, stat]. [22] Y. Lin, Z. Liu, M. Sun, Y. Liu, X. Zhu, Learning Entity and Relation Embeddings for Knowledge Graph Completion, Proceedings of AAAI 29 (2015) 2181–2187. doi:10.1609/ aaai.v29i1.9491. [23] K. Hayashi, K. Kishimoto, M. Shimbo, Binarized embeddings for fast, space-eficient knowledge graph completion, IEEE Transactions on Knowledge and Data Engineering 35 (2021) 141–153.