Design and development of machine learning approaches for the domain of recommendation systems revolves around the interplay between scalability and approximation capability of classification and regression algorithms. Currently, many deployed recommendation engines rely on factorization machine-based approaches; this is mostly due to good trade-offs when it comes to scalability, maintainability and data scientists' involvement in building such models. Even though contemporary recommenders started to increasingly rely on language model-based techniques [1], utilizing factorization machines remains de facto solution for large-scale "screening" of candidates that are to be served. Such candidates can include from unseen items (online stores), to movie recommendations, to ads [2, 3]. Scalability of factorization machines enables creation of real-time systems that handle hundreds of millions of requests in predictable and maintainable manner. In recent years, two main branches of methods have emerged. Approaches based on frameworks such as TensorFlow [4] and PyTorch [5] enabled construction of highly expressive architectures that often require specialized hardware for efficient productization [6, 7, 8, 9]. CPUonly, single instance -single pass alternatives are fewer, and revolve around highly optimized C++ or Rust-based approaches that exploit consumer hardware as much as possible. The latter is the main focus of this paper (overview in Figure 1).
We proceed with a discussion of Fwumious Wabbit (FW), an in-house, Rust-based factorization machine-based system currently used in production for large-scale recommendation 1 .
The FW derives from Vowpal Wabbit (VW) [10], a highperformance, scalable open-source ML system recognized for its efficiency on large datasets 2 . While VW primarily uses logistic regression for tasks like click-through rate prediction, it lacks readily available advanced extensions found in the domain of factorization machines. One of the more expressive variations of factorization machines are the Field-aware Factorization Machines (FFMs), described in detail in the works of Juan et al. [11, 12]. Building on this foundation, we enhanced the FFM architecture by integrating elements of deep learning. Specifically, a multi-layer perceptron (MLP)-like structure in conjunction with the traditional FFM (and logistic regression) components. The architecture's computational complexity, a notable challenge, contributes to its rarity in existing benchmarks. When implemented in standard frameworks like TensorFlow, the architecture struggles to scale effectively for practical use. Despite these challenges, our deep learning-extended FFM method demonstrated significant performance gains over other tested algorithms in internal assessments. However, scaling this method was not straightforward. It was only through invoking BLAS [13], that we achieved critical performance enhancements, allowing for practical full-scale deployment 3 . An overview of the architecture is shown in Figure 2. . Key parts of the architecture are
takes as input both FFM and LR's outputs, i.e. dffm(W 1,2,…,𝑛 , w 𝑏 , w 𝑐 , x) =ffnn(W 1,2,…,𝑛 , 𝑀𝑒𝑟𝑔𝑒𝑁 𝑜𝑟𝑚𝐿𝑎𝑦𝑒𝑟 (lr(w 𝑏 , 𝑥), 𝐷𝑖𝑎𝑔𝑀𝑎𝑠𝑘(ffm(w 𝑐 , 𝑥))).
Here, MergeNormLayer represents the operator that combines outputs of FFM and LR parts and applies normalization. Further, DiagMask represents diagonal mask of FFM space, inducing half smaller number of combinations requiring down-stream processing4 .
Even though we evaluated FW extensively on internal data sets (and online, in A/B tests), where it showed consistent dominance, results on published data sets such as Criteo are also of relevance for dissemination of engines' behavior and overall performance. In this section we overview a benchmark we conducted to assess general behavior of VW and FW. We also implemented DCNv2 [14, 15], a Tensorflowbased strong baseline 5 . For considered data sets (Criteo6 , Avazu7 and KDD20128 ), log transform of continuous features was conducted and no additional data pruning (rare values etc.) was conducted (as is done in our system) 9 . The hyperparameters considered include power of t, learning rates for different types of blocks (ffm, lr), regularization amount (L2 norm, VW). For DCNv2 we considered different learning rates, cross layer numbers, dropout rates and beta parameters. Results of the benchmark are summarized in Figure 3. For each data set, algorithms considered are visualized as AUC scores computed in a rolling window of 30k instances 10 . The trace in each plot represents the average performance (95% CI), and light-gray regions represent model evaluations that were out-of-distribution -this aspect is particularly relevant for understanding stability of different approaches and their sensitivity to hyperparameter configurations. For example, we observed that adding deep layers to VW models in most cases resulted in worse performance. Carefully tuned VW hyperparameters yielded sufficient performance, however, indicate potentially cumbersome model search (when considering new use cases/data) in practice. Similar behavior was observed for DCNv2. The dotted black lines represent the overall best single-window performance, and performance on a given data set's test set 11 Overall, initial phases of learning revealed VW's capability to adapt with less data, the DeepFFMs dominate after enough data is seen by the engines. Superior performance was observed by DCNv2 on Criteo, yet not other data sets (all features considered). The benchmark demonstrates that progressively more complex architectures tend to result in better modeling capabilities, and with them, better AUCs in this benchmark. In terms of runtime, on the same hardware, Criteo data set could be processed on average in 32min by VW, and 31min by FW (linear model vs. DeepFFM). Deep VW variations took substantially longer, around 65min on average (batch size of 2k). This result indicates that FW enables more powerful models with same time bounds for training. The DCNv2 (CPU) baseline was 30%-50% slower compared to DeepFFM runs. These tatistics were obtained based on tens of thousands of runs that represented different algorithm configurations (both hyperparameters and field specifications). Being CPU-based, the described approaches enable seamless scaling to commodity hardware, resulting in lower training and inference costs in practice.
This section aims to facilitate understanding of subsequently discussed optimizations that were put in place to enable scaling of Deep FFMs. The implemented FW contains both training and inference logic. The training logic is relevant for incrementally training more than a hundred models, online, every 𝑛 minutes (depends on the model). Training jobs are separate deployments that automatically query for relevant chunks of data, download, update based on existing weights and send the weights to the serving layer. Serving layer on-the-fly reconstructs the final inference weights via a patching mechanism discussed in Section 6, and exposes the weights as part of the serving service that handles millions of requests with new data. Based on the effect of predictions, data is streamed back to the system as training Serving binds the inference capabilities with the serving (Java) service directly via a foreign function interface (ffi) 12 .
The architecture enables separation of concerns -training jobs are separate to inference jobs, albeit at the cost of needing to send the updated weight data between services; this is one of the key performance bottlenecks that was addressed in this work. An overview of the scope of this paper is shown in Figure 1.
We next discuss main improvements implemented at the level of training jobs and offline research.
Model warm-up corresponds to a phase in model training where model starts with past data, and "catches up" with present data as fast as possible. We identified efficient data pre-fetching as a crucial optimization for speeding up this process. By implementing async learning cycles, multiple rounds of "future" data can be downloaded upfront, making sure the learning engine has constant influx of data. Data pre-fetch in practice results in up to 4x faster prewarming. Within the cloud environment where the jobs are deployed, we can control machine "taints", i.e. signatures that determine their hardware profile. Pre-warm jobs have dedicated taints, which in practice results in machines that are newer and stronger.
An optimization that significantly improved model prewarm time is the previously reported Hogwild-based model training [16], implemented also for Fwumious framework (as part of this work). Here, weight overlaps/overrides are allowed as the trade off for multi-threaded updates. By tuning Hogwild capacity to tainted machines, we observed multi-fold speedups in model warm-up. In practice, the times for bigger models went from multiple weeks to days, and in most cases around a day of training (to catch up).
The next discussed optimization is related to how gradients are accounted for during model optimization itself. We observed that deep layers, albeit being parameter-wise in minority compared to FFM part, take up considerable amount of time during optimization. To remedy this shortcoming, we identified an optimization opportunity that is a combination of activation function used in most models, 𝑓 (𝑥) = max(𝑥, 0), and the specific implementation of FW.
By realizing that we can identify zero global gradient scenarios upfront, prior to updating any weights, we could skip whole branches of computation with no impact on learning.
The performance (speed) of training however, was acrossthe-board improved by 30% for most models, and for deeper ones by up to 3x, see Table 3 for more details. We observed that at most two hidden layers were feasible for production, hence any further speedups than observed 30% were not feasible in practice. This optimization was possible due to ReLU's nature; this activation maps weights to zeros, effectively enabling identification of compute branches that need to be skipped during updates.
We proceed our discussion with an overview of CPU-based model inference via context caching. A considerable optimization we observed could take place in our system is context caching. Each request can be separated into context and candidates. For all candidates in the request, the context is the same, even though the recommended content's features differ -this implies part of the feature space is very consistent for each candidate batch. To exploit this property, a dedicated serving-level caching scheme was put in place. FW at this point does an additional pass only with the context part, where it identifies and caches frequent parts of the context. On subsequent candidate passes it reuses this information on-the fly instead of re-calculating it for each context-candidate pair. Deployment impact of context caching is shown in Figure 4 13 . We next discuss (SIMD) Instruction-aware forward pass. Another optimization that is particular to inference is proper exploitation of SIMD intrinsics. These hardware instruction level optimizations, however, needed to be carefully implemented as the space of serving hardware is not homogeneous, meaning that onthe-fly instruction detection, and subsequent utilization of appropriate binary needed to be put in place. SIMD intrinsics were successfully used to speed up forward pass 13 https://github.com/outbrain/fwumious_wabbit/blob/main/src/radix_ tree.rs (inference) with no loss in RPM performance, and resulted in a consistent 20% speedup for all serving 14 . Real-life example of deployed SIMD-based FW vs. the control (no SIMD) is shown in Figure 5. Up to 25% faster inference (and with it lower resource utilization) were observed.
As discussed in previous sections, training and serving jobs are separated. This separation of concerns, albeit easier to maintain, contributes to a major drawback: weight sending across the network. Model weights need to be constantly updated, which incurs substantial bandwidth costs. For example, hundreds of live models that take up to 10G of memory (per update) are constantly transferred across the network, resulting in a substantial bandwidth overhead to ensure low-latency online serving. Model patching. The first improvement we implemented is the concept of model patching. This process is inspired by application of software patches (in general), albeit tailored to internal structure of FW's weights. Each trained model consists of training weights and the optimizer's weights. The latter are not required for actual inference, which immediately reduces the required space by half. Further, each subsequent inference weights update (inference weights can be multiple GB) first computes model diff -byte-level difference between old and new weights. This is possible due to a consistent memory-level structure of weight files. The diffs are compressed, sent to the serving layer, unpacked and applied to previous weights file to obtain the new set of weights (inference). This process takes tens of seconds, however, further reduces memory footprint on the network by more than 100% (less than a GB of updates per model after patching Deep FFMs).
First, instead of storing absolute indices of bytes that change, relative locations are stored, resulting in a considerable storage saving. Next, small integers denoting these differences are stored as a custom integer type -instead of storing whole ints, compressed versions (small ints are impacted the most) are stored, leading to further improvements 15 . As patcher works at the level of bytes, we also successfully tested it for internal Tensorflow-based flows (reduced bandwidth for sending models). Weight Quantization. Inspired by recent weight quantization advancements in the field of large language models [17, 18], we implemented a variation of 16b weight quantization that, when combined with the byte-level patching mechanism, offered considerable bandwidth and model storage improvements. The quantization algorithm was designed to account for the following use-case specific properties. First, by ensuring consistently small weight patches, the quantization ensures consistently smaller network load. Second, the quantization and dequantization procedures must be fast, as they need to happen within a designated time window after each training round (procedure has tens of seconds at most at its disposal for full weight space). Finally, the algorithm needs to be able to dynamically select viable weight ranges, as we observed considerable variation in weight update sizes based on e.g., time of the day (traffic amount). The final version of the algorithm can be summarized as follows.
For each online model update (e.g., 5min window), weights are first traversed to obtain the minimum and maximum values (weights). These statistics are required to dynamically determine the range of relevant weight bins, as the amount of possible values for 16b representation is small (around 65k). Let 𝑊 = {𝑤 1 , 𝑤 2 , … , 𝑤 𝑛 |𝑤 𝑖 ∈ ℝ} denote the set of all (𝑛) weights and 𝑏 max denote the number of possible weight buckets. Once the minimum and maximum are obtained, the bucket size is computed as bucket 𝑠 = max(𝑊 ).round(𝛼) − min(𝑊 ).round(𝛽) 𝑏 max .
Note that minimum and maximum are rounded to 𝛼 and 𝛽 decimals. This consideration stems from empirical results that indicated that considering full precision bounds results in less stable patch sizes 16 . When constraining minimum and maximum to certain precision, behavior stabilized whilst preserving performance and online behavior. In the second pass, weights are quantized -for each weight, its 16b representation is computed and stored. This results in computing ((𝑤 𝑖 − min(𝑊 )/bucket 𝑠 ).round().castTo16b().convertToBytes(),
i.e. a set of bytes that represent a certain weight bucket. Bytes are stored in FW weight format and re-used during inference. An important detail also concerns metadata required to perform this type of quantization; the original weights file is enriched with a header that contains the bucket size and weight minimum -these two properties are sufficient for efficient weight reconstruction when/where relevant 17 Note that weight patching and quantization on their own already at least halve the size of weights that are used in serving and production. Further, by combining the two approaches, we observed a non-linear improvement in patch sizes -around 10x smaller updates are regularly produced. The quantized patches-based model showed small lifts in and online A/B against control with no quantization applied, considerably reducing network bandwidth required with a small positive business impact (+0.15% RPM). Speedup in a real-life production system due to compound effect of quantization and patching can be observed in Figure 6. Rightmost part of the plot represents total time spent patching and computing quantized weights.
In this paper, we presented a collection of implementation details for scaling CPU-based DeepFFMs to operate at a multi-data-center scale, capable of handling hundreds of millions of predictions per second. We delved into both the offline and online components of our system. In the offline phase, we covered the complete workflow, including model architecture, enhancements to system warm-up processes, and bandwidth optimization strategies. Within the online phase, we describe two novel modifications to the inference layer that have yielded significant speed improvements. Our main algorithms, concepts, and performance benchmarks were discussed in detail, open-source implementations of key components were made freely available. The implementation is extensible to other FFM-based variants. As further work, on the inference side, implementing quantization techniques could accelerate the forward pass by using integer-based operations [19]. Improved weight sharing and memory mapping could offer training improvements.