Artificial intelligence has become an integral part of modern life, with pervasive AI leveraging interconnected devices to deliver intelligent services in real-time environments with the most challenging aspects emerging in the domain of mobile edge computing (MEC). Deploying deep neural networks (DNNs) on such devices and orchestrating the necessary communication require dynamic coordination, an area where reinforcement learning (RL) has shown significant promise. In this paper, we propose a Double Deep Q-Network (DDQN) approach for DNN inference ofloading within a mobile edge computing system, eficiently addressing the challenges of dynamic orchestration while optimizing hierarchical collaboration between edge devices. Our experimental results demonstrate the strength of this approach, achieving superior performance in optimizing inference ofloading compared to existing solutions.
Artificial intelligence (AI) has found applications in numerous domains, ofering transformative solutions
across industries. In particular, deep neural networks (DNNs) have proven efe ctive in tackling complex
challenges, driving innovation in areas such as healthcare [
One key challenge in deploying DNNs in IoT environments is the tradeof between model accuracy and
the computing requirements associated with the limited hardware capabilities of many IoT nodes [
The remainder of this paper is structured as follows: Section 2 provides an overview of related work, Section 3 details our proposed approach, and Section 4 presents the results of our experiments. Finally, we conclude by summarizing our contributions and ofering directions for future research.
Reinforcement learning (RL) has proven to be an efective approach for handling sequential
decisionmaking problems, especially in dynamic environments. This makes it well-suited for managing system
dynamics in DNN inference ofloading. In [
In [
As summarized in Table 1, existing studies commonly address the distributed inference ofloading problem by leveraging reinforcement learning (RL) approaches and their deep variants, which have proven efective in handling complex state spaces. These RL methods involve formulating the problem as a Markov Decision Process (MDP), requiring a well-designed state space, action space, and a reward function aligned with the desired objectives.
These studies can be categorized into policy-based and value-based approaches. Policy-based methods,
such as those employing the actor-critic framework [
The Double DQN algorithm [16] aims to mitigate the overestimation problem found in traditional Q-learning. In standard Q-learning and DQN, the same parameters (the online Q network) are used for both action selection and evaluation, which can result in inflated Q-value estimates, especially for actions consistently given higher values. Double DQN addresses the overestimation issue in traditional Q-learning by introducing "target networks" Instead of using a single set of parameters for both action selection and evaluation, Double DQN uses two separate networks: the online network for selecting actions and the target network for evaluating Q-values. The target network, updated periodically with the online network’s parameters, provides more stable and less optimistic estimates, reducing overestimation during evaluation.
During training, the parameters of the online network are updated by minimizing the loss function, commonly using the Mean Squared Error (MSE) between the Q-value estimated by the online network and the target value generated by the target network. This process begins by randomly sampling state transitions from a replay memory bufer, which holds a collection of past actions and states. Unlike traditional Q-learning that relies on matrix-based operations, these updates are processed by neural networks guided by the Bellman equation.
Figure 1 illustrates the Q-Network and its connection to the replay memory bufer, which stores tuples representing individual experiences. Each tuple contains the current state, the chosen action, the resulting new state after action execution, the observed reward, and an indication of episode termination. These stored experiences are sampled to compute the Bellman equation term using the target network’s inference. The online network is then optimized through gradient descent to approach this target Bellman value. As shown in Figure 1, both the online and target networks share identical architectures. Each network is composed of multilayer perceptrons (MLPs) that receive a flattened representation of the state space (as detailed in Section 3.3) and output value estimates for each possible action.
By decoupling the action selection and evaluation processes, Double DQN reduces the likelihood of overestimating Q-values, leading to more stable and accurate learning. This approach has been shown to improve the performance and training stability of deep reinforcement learning algorithms, particularly in environments with large state spaces or complex dynamics.
We operate in a mobile edge computing (MEC) system comprising multiple edge devices of number , each equipped with a deployed DNN model. All devices are connected to an edge server via wireless communication, which also hosts a DNN model to handle any remaining processing tasks. The DNN models are partitioned into segments, indexed by for each device , with each segment generating a number of floating-point operations denoted as . The partitioning is achieved using cut layers, which in our current work are selected as pooling layers due to their smaller output sizes, facilitating more eficient wireless transmission. When a device performs inference up to a selected cut layer (), the computing delay is directly proportional to the number of layers processed and the device’s hardware characteristics, such as CPU frequency . This delay is determined by the following formula: (1) (2) (3) (4) (5) (6)
Here, 2 represents the spectral noise, while () denotes the time-varying channel gain between the device and the server, capturing the fluctuating network conditions over time. It is important to note that the available bandwidth is equally divided among all devices, which is not the most eficient allocation strategy, as the computing demands of individual devices usually vary. Nevertheless, we adopt this approach since our current focus is on the reinforcement learning method rather than optimizing resource allocation. We will also note that 0 is equal to the initial input size multiplied by the compression rate (). And the transmission energy consumption formula is given by : () = ()
We adopt a linear model to estimate the energy consumption during computation, as outlined in [19]. Given the energy eficiency coeficient the energy consumption is defined as:
For the communication delay, we use Shannon’s formula, which relates the transmission delay to the output data size from the partial inference , the transmission power of the device , and the network bandwidth , resulting in the following formula: () = ∑︀=− 01 ( − ()) × ×
() = () × ( )3 × () =
(1 + () )
2 − 1 (() − ) × × () = ∑︁
=0 () = {︃0, if < 0
1, if >= 0 denotes the device’s computing performance (number of cycles per flop), and is defined as: The remaining computation is carried out on the server side, where the CPU frequency is , as determined by the following formula:
Given arrived input data, and a corresponding selected cut layer (), the total delay is equal to: − 1 ⎛ (() − ) + ( − ()) + () = () ∑︁ ⎝ =0 log(1 + () ) ⎠ 2 ⎞
By presenting our system model, we arrive at the formulation of the optimization problem, which involves:
Minimize
1 ∑︁ ∑︁ () =0 =1
This is achieved by determining the decision variables () and (), while accounting for the stochastic variations in network conditions () and the data arrival rate (). Consequently, solving this problem is NP-hard, necessitating the use of a reinforcement learning approach that sequentially optimizes the decision variables through a learned policy. However, prior to this, we need to model our problem as a Markov Decision Process (MDP), which involves defining the state, action, and reward structures.
• State : We need to monitor the data arrival rate (), the network conditions through (), and the current workloads of both the device and server. These will be represented as the available computing and transmission queues sizes Ψ () and Ψ () respectively, and also the computing queue size for the server (). The partial observations from each device will be consolidated into a global observation. This aggregation will consequently define our state space as follows: () = ( (), (), (), (), ()) • Action : Each device must determine both the cut layer at which to perform data inference and the compression rate to apply. Consequently, the action space will encompass the collective actions of all devices, resulting in: () = ((), ())
() = ∑︁ () =1 (7) (8) (9) with () = (1(), ...(), ... ()) and () = (1(), ..., (), ..., ()) • Reward : Since our objective is to optimize the average service delay, we choose to maximize its dual aspect, known as throughput. In this context, we reward the agent for each data completion within the system, specifically the total number of completed tasks () across all devices:
In our work, we consider a single agent operating on the server side, which manages two Q networks (the online and target networks) and maintains a memory bufer for storing experiences. This agent is responsible for observing the system state by gathering information from the devices and the environment, formatting this data in accordance with Section 3.3, and subsequently determining the actions for each device using an epsilon-greedy policy. This approach involves selecting random actions with a probability of epsilon to encourage exploration, while also enabling exploitation by utilizing the online Q network to estimate the Q values for each action combination, ultimately selecting the action with the highest Q value [20]. Subsequently, the corresponding actions are dispatched to each device, prompting them to execute these tasks. This includes compressing the input data at the specified rate and carrying out DNN inference up to the designated cut layer. Afterward, the devices send the intermediate inference results to the server, which completes the remaining computations and returns the final results to the respective devices. Once this cycle is completed, each device receives a reward based on the formulation outlined in Section 3.3, and the results are transmitted to the server for aggregation. The entire environment then transitions to a new state ′. This experience is recorded in the replay bufer for future use in updating the Q networks. The online Q network is updated once a suficient number of experiences have been gathered, specifically a batch size of stored experiences. This update utilizes the standard Bellman equation, which establishes the relationship between the Q value of the current state and that of the subsequent state ′. The key distinction in the DDQN approach lies in applying this equation using the target network, leading to the following formulation: online(, | ) ∼ (1 − ) · online(, | ) + · (, ) + · ma′ x target(′, ′| ) ︂[ ︂] (10) Backpropagation is then performed to minimize the mean squared error between the calculated value and the stored value. After a predetermined number of iterations, determined by a constant , the parameters of the target network are updated by copying those of the online Q network. To facilitate the agent’s transition from exploration to increased exploitation, the epsilon value will be gradually decreased in a linear manner until it reaches a predetermined minimum. This iterative process will persist for a defined number of training episodes. The algorithm is outlined in Algorithm 1. Algorithm 1 Double DQN (RL module at the server side)
Initialize Q-networks online and target for the server with random weights Initialize replay memory Set target network update frequency Set discount factor Set exploration parameters, , min, decay for each episode do
Observe the initial state from state tracking module for each time step in do
Select action using -greedy policy based on online Distribute the actions to the devices Execute Inference Tasks Receive reward and observe the next state ′ Store (, , , ′, done) in replay memory if length of ≥ replay batch size then
Sample a random batch from replay memory for each sample do
Calculate target Q-values using target and Bellman equation
Update online using backpropagation end for end if end if ← end for end for
if is a multiple of then
Update target weights with online weights epsilon-decay( ,
) We developed a simulated environment consisting of a scenario with three edge devices, characterized by the parameters detailed in Table 2. In this setup, the VGG16 classification model is deployed on either the devices or the server. We designated three distinct cut layers—specifically, layers 3, 10, and 18—corresponding to distributed pooling layers, and simulated data collection by randomly sampling from a subset of the ImageNet dataset [21], which consists of 500 images. For the implementation of the classification model, we utilized TensorFlow [22], which also facilitated the collection of intermediate output sizes and the number of floating-point operations (FLOPs) through the TensorFlow Profiler. The agent was designed to comply with the Gymnasium interface [23], while the DDQN algorithm was implemented using the OpenAI Stable-Baselines3 framework [24].
Figure 2a represents the evolution of rewards over successive episodes. Despite some noise, the trend shows a clear upward trajectory as the episodes progress, indicating that the reinforcement learning agent is successfully improving its policy, resulting in consistently higher rewards. While Figure 2b illustrates the evolution of the loss function, showing a decreasing trend. This indicates that the neural networks are successfully training and adapting well to the accumulated experiences. Through these experiments, we demonstrated the convergence and learning progress of the DDQN algorithm.
In this section, we aim to evaluate the performance of our reinforcement learning approach compared
to static configurations, as well as to benchmark it against existing state-of-the-art RL solutions.
We use several performance metrics, including the evolution of the episodic cumulative reward, the
latency—represented by the average service delivery delay derived from the throughput (i.e., the number
of completed inference tasks per episode)—and energy consumption resulting from computation and
transmission. The visible plots represent values averaged over 10 episodes. To achieve the first objective,
we compare our approach with two benchmarks: "edge," where the inference is performed entirely
on the devices, and "central," where the inference is fully ofloaded to the edge server. For the second
objective, we benchmark against the actor-critic approach (A2C), commonly used in state-of-the-art
works [
One notable observation is the marginal superiority of the central approach over the edge solution in terms of system throughput. However, this advantage is contingent upon favorable network conditions, where server performance in computing becomes the guiding factor. As anticipated, the edge solution exhibits stable behavior across varying network conditions. Furthermore, the reinforcement learning solutions (DDQN, DQN and A2C) demonstrate superior adaptability after suficient training episodes, outperforming the former approaches (edge and central).
For the comparison of reinforcement learning approaches, the results demonstrate that the policy-free methods (DQN and DDQN) both outperform the A2C approach in terms of learning behavior, showing greater stability and higher reward values. Additionally, these approaches achieve better performance metrics, such as increased throughput, which leads to reduced latency. Moreover, the DDQN approach exhibits more stability compared to DQN, as observed in the results. This highlights the superiority of the DDQN algorithm over other deep reinforcement learning methods. This experiment serves as empirical validation of the findings in [ 16] within the context of DNN inference ofloading, showing that DDQN improves Q-value estimation stability by reducing overestimation bias.
In this work, we tackled the challenging problem of DNN inference ofloading using a reinforcement learning approach, specifically leveraging the DDQN algorithm due to its robust capabilities. Our results demonstrated its competitive performance against state-of-the-art solutions. Future research could focus on refining the MDP design by incorporating additional state and action elements, enhancing the system’s adaptability. Additionally, shaping the reward function to encompass a broader range of objectives, such as more eficient resource management or explicit energy optimization, ofers promising directions for further improving the system’s performance.
During the preparation of this work, the authors used OpenAI’s ChatGPT (GPT-4-turbo, February 2024 version) in order to: Grammar and spelling check, as well as language refinement. After using this service, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. [16] H. Van Hasselt, A. Guez, D. Silver, Deep reinforcement learning with double q-learning, in:
Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016. [17] Q. Zhang, M. Lin, L. T. Yang, Z. Chen, S. U. Khan, P. Li, A double deep q-learning model for energy-eficient edge scheduling, IEEE Transactions on Services Computing 12 (2019) 739–749. doi:10.1109/TSC.2018.2867482. [18] A. Iqbal, M.-L. Tham, Y. C. Chang, Double deep q-network-based energy-eficient resource allocation in cloud radio access network, IEEE Access 9 (2021) 20440–20449. doi:10.1109/ ACCESS.2021.3054909. [19] A. Beloglazov, R. Buyya, Optimal online deterministic algorithms and adaptive heuristics for energy and performance eficient dynamic consolidation of virtual machines in cloud data centers, Concurrency and Computation: Practice and Experience 24 (2012). URL: https://api.semanticscholar. org/CorpusID:10061036. [20] O. Berger-Tal, J. Nathan, E. Meron, D. Saltz, The exploration-exploitation dilemma: a multidisciplinary framework, PloS one 9 (2014) e95693. [21] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., Imagenet large scale visual recognition challenge, International journal of computer vision 115 (2015) 211–252. [22] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL: https://www.tensorflow.org/, software available from tensorflow.org. [23] M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goulão, A. Kallinteris, M. Krimmel, A. KG, et al., Gymnasium: A standard interface for reinforcement learning environments, arXiv preprint arXiv:2407.17032 (2024). [24] A. Rafin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, M. Chevalier-Boisvert, J. Kubricht, A. Nichol, B. Hill, J. Pachocki, et al., Stable baselines3, https://github.com/DLR-RM/ stable-baselines3, 2020.