Field-aware Factorization Machines (FFMs) have emerged as a powerful model for click-through rate prediction, particularly excelling in capturing complex feature interactions. In this work, we present an in-depth analysis of our in-house, Rust-based Deep FFM implementation, and detail its deployment on a CPU-only, multi-data-center scale. We overview key optimizations devised for both training and inference, demonstrated by previously unpublished benchmark results in eficient model search and online training. Further, we detail an in-house weight quantization that resulted in more than an order of magnitude reduction in bandwidth footprint related to weight transfers across data-centres. We disclose the engine and associated techniques under an open-source license to contribute to the broader machine learning community. This paper showcases one of the first successful CPU-only deployments of Deep FFMs at such scale, marking a significant stride in practical, low-footprint click-through rate prediction methodologies.
and storage
Path to production TensorFlow 4[] and PyTorch [5] enabled construction of highly expressive architectures that often require specialized hardware for eficient productization6,[7, 8, 9]. CPUonly, single instance – single pass alternatives are fewer, and revolve around highly optimized C++ or Rust-based approaches that exploit consumer hardware as much as possible. The latter is the main focus of this paper (overview in Figure1).
tion1.
We proceed with a discussion of Fwumious Wabbit (FW), an in-house, Rust-based factorization machine-based system currently used in production for large-scale recommenda2.1. Origins of FW and Vowpal Wabbit (VW) The FW derives from Vowpal Wabbit (VW)10[], a highperformance, scalable open-source ML system recognized
overview Performance optimizations that span model search (AutoML), online model training, storage, transfer and serving are discussed.
Design and development of machine learning approaches for the domain ofrecommendation systems revolves around the interplay between scalability and approximation capa- for its eficiency on large dataset2s. While VW primarbility of classification and regression algorithms. Currently, ily uses logistic regression for tasks like click-through rate many deployed recommendation engines rely on factoriza- prediction, it lacks readily available advanced extensions tion machine-based approaches; this is mostly due to good trade-ofs when it comes to scalability, maintainability and data scientists’ involvement in building such models. Even the Field-aware Factorization Machines (FFMs), described though contemporary recommenders started to increasingly in detail in the works of Juan et a1l1., [12]. Building on this found in the domain of factorization machines. One of the more expressive variations of factorization machines are rely on language model-based technique1s],[utilizing factorization machines remaindse facto solution for large-scale ”screening” of candidates that are to be served. Such candi-perceptron (MLP)-like structure in conjunction with the tradates can include from unseen items (online stores), to movie ditional FFM (and logistic regression) components. The arrecommendations, to ads2[, 3]. Scalability of factorization machines enables creation of real-time systems that handle contributes to its rarity in existing benchmarks. When imhundreds of millions of requests in predictable and maintain-plemented in standard frameworks like TensorFlow, the able manner. In recent years, two main branches of methods architecture struggles to scale efectively for practical use.
chitecture’s computational complexity, a notable challenge, have emerged. Approaches based on frameworks such as foundation, we enhanced the FFM architecture by integrating elements of deep learning. Specifically, a multi-layer AdKDD Workshop 2024 ∗Corresponding author. 0000-0002-9916-8756 (B. Škrlj) CEUR
ceur-ws.org
Despite these challenges, our deep learning-extended FFM method demonstrated significant performance gains over other tested algorithms in internal assessments. However, scaling this method was not straightforward. It was 1The engine with main implementations discussed in this paper is freely
instance1s0. hyperparameters considered include power of t, learning rates for diferent types of blocks (fm, lr), regularization amount (L2 norm, VW). For DCNv2 we considered diferent learning rates, cross layer numbers, dropout rates and beta parameters. Results of the benchmark are summarized in alized as AUC scores computed in a rolling window of 30k
The trace in each plot represents the average performance (95% CI), and light-gray regions represent model evaluations that were out-of-distribution – this aspect is particularly relevant for understandsintagbility of diferent approaches and their sensitivity to hyperparameter configurations. For example, we observed that adding deep layers to VW models in most cases resulted in worse performance. Carefully tuned VW hyperparameters yielded suficient performance, however, indicate potentially cumbersome model search (when considering new use cases/data) in practice. Similar behavior was observed for DCNv2. The dotted black and performance on a given data set’s test11seOtverall, initial phases of learning revealed VW’s capability to adapt with less data, the DeepFFMs dominate after enough data is seen by the engines. Superior performance was observed by DCNv2 on Criteo, yet not other data sets (all features considered). The benchmark demonstrates that progressively more complex architectures tend to result in better modeling capabilities, and with them, better AUCs in this benchmark. In terms ofruntime, on the same hardware, Criteo data set could be processed on average in 32min by VW, and 31min by FW (linear model vs. DeepFFM). Deep VW variations took substantially longer, around 65min on average (batch size of 2k). This result indicates that FW enables more powerful models with same time bounds for training. The DCNv2 (CPU) baseline was 30%-50% slower compared to DeepFFM runs. These tatistics were obtained LR performance enhancements, allowing for practical full-scalelines represent the overall best single-window performance, down-stream processin4g.
Here,MergeNormLayer represents the operator that com- based on tens of thousands of runs that represented diferent bines outputs of FFM and LR parts and applies normalization. algorithm configurations (both hyperparameters and field FurtherD,iagMask represents diagonal mask of FFM space,
specifications). Being CPU-based, the described approaches inducing half smaller number of combinations requiring enable seamless scaling to commodity hardware, resulting in lower training and inference costs in practice.
Even though we evaluated FW extensively on internal data
values etc.) was conducted (as is done in our syst9e.mT)he sets (and online, in A/B tests), where it showed consistent This section aims to facilitate understanding of subsequently dominance, results on published data sets such as Criteo are discussed optimizations that were put in place to enable also of relevance for dissemination of engines’ behavior and scaling of Deep FFMs. The implemented FW contains both overall performance. In this section we overview a bench- training and inference logic. The training logic is relevant mark we conducted to assess general behavior of VW and
for incrementally trainimngore than a hundred models, FW. We also implemented DCNv2 1[4, 15], a Tensorflowbased strong baselin5e. For considered data sets (Crit6,eo online, ever y minutes (depends on the model). Training jobs are separate deployments that automatically query for Avazu7 and KDD20128), log transform of continuous fea- relevant chunks of data, download, update based on existing tures was conducted and no additional data pruning (rareweights and send the weights to the serving layer. Serving layer on-the-fly reconstructs the final inference weights via a patching mechanism discussed in Section 6, and ex3https://github.com/outbrain/fwumious_wabbit/blob/main/src/block_ poses the weights as part of the serving service that handles https://github.com/outbrain/fwumious_wabbit/blob/main/src/ neural.rs 4See regressor.rsfor more details. implementation. 5Unique hash was assigned to each value for this baseline for ease of 6https://www.kaggle.com/c/criteo-display-ad-challenge 7https://www.kaggle.com/c/avazu-ctr-prediction/data 8https://www.kaggle.com/c/kddcup2012-track2 9Such minimal pre-processing is within reach of a regular production.
publications. millions of requests with new data. Based on the efect of predictions, data is streamed back to the system as training 10RIG and Log-loss scores are aligned with AUC-based results, hence
only these are reported for readability purposes 11for KDD, we took last 2m instances to capture apparent variability in data better, other data sets are split as reported in their origin 0.8 0.7 0.6 0.5 0.8 0.7 0.6 0.5 0.8 0.7 0.6 0.5 0 0 0 2 4 #inst. (VW-linear) ×107
2 4 #inst. (VW-mlp) ×107
2 4 #inst. (FW-DeepFFM)×107
2 4 #inst. (FW-FFM) ×107
2 4 #inst. (DCNv2) ×107 2 4 #inst. (VW-linear) ×107
2 4 #inst. (VW-mlp) ×107
2 4 #inst. (FW-DeepFFM)×107
2 4 #inst. (FW-FFM) ×107
2 4 #inst. (DCNv2) ×107 0.8 0.7 0.6 0.5 0.8 0.7 0.6 0.5 0.8 0.7 0.6 0.5 0 0 0 0.8 0.7 0.6 0.5 0.8 0.7 0.6 0.5 0.8 0.7 0.6 0.5 0 0 0 ) 0.8 k 0 s3=0.7 w ( AUC0.6 0.5 ) 0.8 k 0 s3=0.7 w ( AUC0.6 0.5 ) 0.8 k 0 s3=0.7 w ( AUC0.6 0.5 0 0 0 0.8 0.7 0.6 0.5 0.8 0.7 0.6 0.5 0.8 0.7 0.6 0.5 0 0 0 1 2 #inst. (VW-linear) ×107
1 2 #inst. (VW-mlp) ×107
1 2 #inst. (FW-DeepFFM)×107
1 2 #inst. (FW-FFM) ×107
1 2 #inst. (DCNv2) ×107
Model warm-up corresponds to a phase in model training where model starts with past data, and ”catches up” with present data as fast as possible. We identified eficient data pre-fetching as a crucial optimization for speeding up this process. By implementing async learning cycles, multiple rounds of ”future” data can be downloaded upfront, making sure the learning engine has constant influx of data. Data pre-fetch in practice results in u4pxtfoaster prewarming. Within the cloud environment where the jobs are deployed, we can control machine ”taints”, i.e. signatures that determine their hardware profile. Pre-warm jobs have dedicated taints, which in practice results in machines that are newer and stronger. 12https://github.com/outbrain/fwumious_wabbit/blob/main/src/lib.rs
The next discussed optimization is related to how gradients are accounted for during model optimization itself.Figure 4: Impact of context caching on inference time. We observed that deep layers, albeit being parameter-wise in minority compared to FFM part, take up considerable amount of time during optimization. To remedy this shortcoming, we identified an optimization opportunity that is a combination of activation function used in most models, () = max (, 0), and the specific implementation of FW.
By realizing that we can identzifeyro global gradient scenarios upfront, prior to updating any weights, we could skip whole branches of computation with no impact on learning.
The performance (speed) of training however, was acrossthe-board improved by 30% for most models, and for deeper ones by up to 3x, see Table3 for more details. We observed Figure 5: Relative impact of SIMD-enabled (blue, after drop) vs. that at most two hidden layers were feasible for productionS,IMD-disabled (purple) FW in production (inference). hence any further speedups than observed 30% were not feasible in practice. This optimization was possible due to (inference) with no loss in RPM performance, and resulted ReLU’s nature; this activation maps weights to zeros, efec- in a consistent 20% speedup for all serv1i4n. gReal-life examtively enabling identification of compute branches that need ple of deployed SIMD-based FW vs. the control (no SIMD) to be skipped during updates. is shown in Figure5. Up to 25% faster inference (and with it lower resource utilization) were observed.
We proceed our discussion with an overview of CPU-based model inference viacontext caching. A considerable op- As discussed in previous sections, training and serving jobs timization we observed could take place in our system is are separated. This separation of concerns, albeit easier to context caching. Each request can be separated into context maintain, contributes to a major drawback: weight sending and candidates. For all candidates in the request, the con-across the network. Model weights need to be constantly text is the same, even though the recommended content’s updated, which incurs substantial bandwidth costs. For features difer – this implies part of the feature space is very example, hundreds of live models that take up to 10G of consistent for each candidate batch. To exploit this prop-memory (per update) are constantly transferred across the erty, a dedicated serving-level caching scheme was put in network, resulting in a substantial bandwidth overhead to place. FW at this point does an additional pass only with the ensure low-latency online serving. context part, where it identifies and caches frequent parts Model patching. The first improvement we impleof the context. On subsequent candidate passes it reuses mented is the concept omfodel patching. This process is this information on-the fly instead of re-calculating it for inspired by application of software patches (in general), aleach context-candidate pair. Deployment impact of context beit tailored to internal structure of FW’s weights. Each caching is shown in Figure413. We next discuss(SIMD) trained model consists of training weights and the optiInstruction-aware forward pass. Another optimization mizer’s weights. The latter are not required for actual inthat is particular to inference is proper exploitation of SIMDference, which immediately reduces the required space by intrinsics. These hardware instruction level optimizations,half. Further, each subsequent inference weights update however, needed to be carefully implemented as the space (inference weights can be multiple GB) first computmeosdel of serving hardware is not homogeneous, meaning that on- dif – byte-level diference between old and new weights. the-fly instruction detection, and subsequent utilization ofThis is possible due to a consistent memory-level structure appropriate binary needed to be put in plaScIeM.D in- of weight files. The difs are compressed, sent to the servtrinsics were successfully used to speed up forward pass ing layer, unpacked and applied to previous weights file 13https://github.com/outbrain/fwumious_wabbit/blob/main/src/radix_ to obtain the new set of weights (inference). This process tree.rs takes tens of seconds, however, further reduces memory footprint on the network by more than 100% (less than a GB of updates per model after patching Deep FFMs).
First, instead of storing absolute indices of bytes that 14https://github.com/outbrain/fwumious_wabbit/blob/main/src/block_ ffm.rs change,relative locations are stored, resulting in a considerable storage saving. Next, small integers denoting these diferences are stored as a custom integer type – instead of storing whole ints, compressed versions (small ints are impacted the most) are stored, leading to further improvements15. As patcher works at the level of bytes, we also successfully tested it for internal Tensorflow-based flows (reduced bandwidth for sending modelWs).eight Quantization. Inspired by recent weight quantization advancements in the field of large language model1s7[, 18], we implemented avariation of 16b weight quantization that, when combined with the byte-level patching mechanism, ofered considerable bandwidth and model storage Figure 6: Speedup observed when jointly using quantization and improvements. The quantization algorithm was designed to model patching (as opposed to just patching). account for the following use-case specific properties. First, by ensuring consistently small weight patches, the quantization ensures consistently smaller network load. Second, theTable 4 quantization and equantization procedures must be fast, Impact of model quantization on the global production CTR as they need to happen within a designated time window model. Weight processing Avg. time spent Update file size after each training round (procedure has tens of seconds at no procecssing (baseline) / 100% most at its disposal for full weight space). Finally, the algo- fw-quantization 2s 50% rithm needs to be able to dynamically select viable weight fw-patcher 45s 30±5% ranges, as we observed considerable variation in weight up- fw-patcher + fw-quantization 8s 3±2% date sizes based on e.g., time of the day (trafic amount). The ifnal version of the algorithm can be summarized as follows. Note that weight patching and quantization on their own For each online model update (e.g5.m,in window), weights already at least halve the size of weights that are used in are first traversed to obtain the minimum and maximum val- serving and production. Further, by combining the two apues (weights). These statistics are required to dynamically proaches, we observed a non-linear improvement in patch determine the range of relevant weight bins, as the amount sizes – around10x smaller updates are regularly produced. of possible values for 16b representation is small (around The quantized patches-based model showed small lifts in 65k). Let = { 1, 2, … , | ∈ ℝ} denote the set of all and online A/B against control with no quantization applied, ( ) weights and max denote the number of possible weight considerably reducing network bandwidth required with buckets. Once the minimum and maximum are obtained, a small positive business impact+(0.15% RPM). Speedup the bucket size is computed as in a real-life production system due to compound efect of quantization and patching can be observed in Fig6u.re max ( ). round() − min( ). round() Rightmost part of the plot represents total time spent patchbucket = . ing and computing quantized weights.
max Note that minimum and maximum arerounded to and decimals. This consideration stems from empirical re- 7. Conclusions and open problems sults that indicated that considering full precision boundIsn this paper, we presented a collection of implementation results in less stable patch si1z6e.sWhen constraining minimum and maximum to certain precision, behavior stabilized details for scaling CPU-based DeepFFMs to operate at a whilst preserving performance and online behavior. In the mmiullltioi-ndsaotfa-pcreendticetriosncsalpee,rcsaepcaobnlde.oWf ehdaneldvleindginhtuonbdortehdtshoef second pass, weights arequantized – for each weight, its 16b representation is computed and stored. This results in ofline and online components of our system. In the ofline computing phase, we covered the complete workflow, including model architecture, enhancements to system warm-up processes, (( − min( )/ bucket ).round().castTo16b().convertToByte()s, and bandwidth optimization strategies. Within the online phase, we describe two novel modifications to the inference i.e. a set of bytes that represent a certain weight bucketl.ayer that have yielded significant speed improvements. Our Bytes are stored in FW weight format and re-used during main algorithms, concepts, and performance benchmarks inference. An important detail also concemrentasdata re- were discussed in detail, open-source implementations of quired to perform this type of quantization; the originalkey components were made freely available. The impleweights file is enriched with a header that contains the mentation is extensible to other FFM-based variants. As bucket size and weight minimum – these two properties are further work, on the inference side, implementing quantizasuficient for eficient weight reconstruction when/where tion techniques could accelerate the forward pass by using relevan1t7. Results on a representative CTR model are integer-based operations19[]. Improved weight sharing shown in Table4. Metrics of interest are time to produce and memory mapping could ofer training improvements. patch and the final patch/weight update’s size. Patching and quantization result in up to 30x smaller model updates. References 15https://github.com/outbrain/fwumious_wabbit/blob/main/weight_
patcher 16(quantization output tended to fluctuate more) 17https://github.com/outbrain/fwumious_wabbit/blob/main/src/ quantization.rs ings of the 17th ACM Conference on Recommender ling, G. Henry, et al., An updated set of basic linear Systems, 2023, pp. 993–999. algebra subprograms (blas), ACM Transactions on [2] S. Zhang, Y. Tay, L. Yao, A. Sun, C. Zhang, Deep Mathematical Software 28 (2002) 135–151. learning for recommender systems, in: Recommender [14] R. Wang, R. Shivanna, D. Cheng, S. Jain, D. Lin, Systems Handbook, Springer, 2021, pp. 173–210. L. Hong, E. Chi, Dcn v2: Improved deep & cross net[3] Y. Deldjoo, M. Schedl, P. Cremonesi, G. Pasi, Recom- work and practical lessons for web-scale learning to mender systems leveraging multimedia content, ACM rank systems, in: Proceedings of the web conference Computing Surveys (CSUR) 53 (2020) 1–38. 2021, 2021, pp. 1785–1797. [4] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, [15] W. Shen, Deepctr: Easy-to-use,modular and extendible C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, package of deep-learning based ctr modehlts,tps:// S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Is- github.com/shenweichen/deepct,2r017. ard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Lev- [16] B. Recht, C. Re, S. Wright, F. Niu, Hogwild!: A lock-free enberg, D. Mané, R. Monga, S. Moore, D. Murray, approach to parallelizing stochastic gradient descent, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, Advances in neural information processing systems K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, 24 (2011).
F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, [17] B. Rokh, A. Azarpeyvand, A. Khanteymoori, A compreM. Wicke, Y. Yu, X. Zheng, TensorFlow: Large-scale hensive survey on model quantization for deep neural machine learning on heterogeneous systems, 2015. networks, arXiv preprint arXiv:2205.07877 (2022). URL: https://www.tensorflow.or,sgo/ftware available [18] H. Bai, L. Hou, L. Shang, X. Jiang, I. King, M. R. Lyu, from tensorflow.org. Towards eficient post-training quantization of pre[5] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Brad- trained language models, Advances in Neural Inforbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, mation Processing Systems 35 (2022) 1405–1418. L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, [19] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, A. Howard, H. Adam, D. Kalenichenko, Quantization L. Fang, J. Bai, S. Chintala, Pytorch: An im- and training of neural networks for eficient integerperative style, high-performance deep learning arithmetic-only inference, in: Proceedings of the IEEE library, in: Advances in Neural Information Pro- conference on computer vision and pattern recognicessing Systems 32, Curran Associates, Inc., 2019, tion, 2018, pp. 2704–2713. pp. 8024–8035. URL: http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library.
pdf. [6] W. Song, C. Shi, Z. Xiao, Z. Duan, Y. Xu, M. Zhang,
J. Tang, Autoint: Automatic feature interaction learning via self-attentive neural networks, in: Proceedings of the 28th ACM international conference on information and knowledge management, 2019, pp. 1161–1170. [7] J. Lian, X. Zhou, F. Zhang, Z. Chen, X. Xie, G. Sun, xdeepfm: Combining explicit and implicit feature interactions for recommender systems, in: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 2018, pp.
1754–1763. [8] H.-T. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye, G. Anderson, G. Corrado, W. Chai, M. Ispir, et al., Wide & deep learning for recommender systems, in: Proceedings of the 1st workshop on deep learning for recommender systems, 2016, pp. 7–10. [9] H. Guo, R. Tang, Y. Ye, Z. Li, X. He, Deepfm: a factorization-machine based neural network for ctr prediction, arXiv preprint arXiv:1703.04247 (2017). [10] A. Bietti, A. Agarwal, J. Langford, A contextual bandit bake-of, arXiv:1802.04064v3 [stat.ML], 2018.
URL: https://www.microsoft.com/en-us/research/ publication/a-contextual-bandit-bake-of.f-2/ [11] Y. Juan, D. Lefortier, O. Chapelle, Field-aware factorization machines in a real-world online advertising system, in: Proceedings of the 26th International Conference on World Wide Web Companion, 2017, pp.
680–688. [12] Y. Juan, Y. Zhuang, W.-S. Chin, C.-J. Lin, Field-aware factorization machines for ctr prediction, in: Proceedings of the 10th ACM conference on recommender systems, 2016, pp. 43–50. [13] L. S. Blackford, A. Petitet, R. Pozo, K. Remington, R. C.
Whaley, J. Demmel, J. Dongarra, I. Duf, S. Hammar