The concept of service migration holds significant importance within the realm of Fog-Edge computing, particularly in scenarios where mobile users are in constant motion, transitioning between various Access Points (APs). While this dynamic mobility is a fundamental characteristic of modern networking environments, it introduces the challenge of the frequent migration of services. The latter can potentially lead to a degradation in the Quality of Service (QoS) experienced by users. This paper addresses this critical issue by presenting a methodical Reinforcement Learning framework for necessary workflow migration in Fog-Cloud Computing. Firstly, we examine the literature solutions and then, we introduce our Markov Decision Model (MDP) for workflow migration.
User mobility in Fog and Edge environments refers to scenarios where users are constantly
moving within the network, such as in a smart city or Internet of Things (IoT) deployment. As
users move, their proximity to Fog nodes may change, and it becomes necessary to migrate
services to Fog nodes that are closer to the users to provide seamless connectivity and optimal
performance [
However, it is important to highlight that frequent migration could result in added migration
expenses, including delays and increased energy usage. Hence, there exists a need to minimize
the frequency of migrations while still adhering to users’ Quality of Service (QoS) requirements,
such as diminishing latency as perceived by users [
Recently, researchers have shown an increasing interest in leveraging artificial intelligence
techniques to propose intelligent solutions for service migration problems in Fog and Edge
environments [
The rest of the paper is organized as follows: In Section 2, we discuss the problem statement, highlighting certain limitations in existing migration solutions that motivated our approach. Section 3 provides an overview of the system modeling. In Section 4, we explain our MDP modeling and RL framework.
Serval works were proposed to solve the service migration problem in Fog-Edge Computing
[
In many existing research studies [8, 10, 9, 12], the primary focus revolves around exploring migration strategies that relocate services or workflows whenever users change locations. However, it’s essential to recognize that such migration strategies can introduce computational resource overhead, higher communication costs, and longer migration times. These unnecessary migrations can deplete resources and disrupt service execution, potentially leading to suboptimal performance and operational ineficiencies.
Conversely, studies aiming to reduce migration frequency [
In our study, depicted in Fig. 4.2, we examine a typical industrial scenario where a robot traverses geographic regions covered by Fog servers. These servers connect to Access Points (APs) through wireless links. Initially, the robot assigns computation tasks to Fog resources in the first region. However, as the user moves, the system must make informed decisions about task migration. These decisions consider factors like resource performance and network conditions. To accommodate user mobility, the system operates in time slots, with the timeline represented by ∈ = {0, 1, 2, . . . , }. The duration of each time slot is (e.g.,15 minutes).
The robot’s cyber workflow consists of tasks with dependencies. The robot’s physical component continuously sends data gathered from various sensors, like images, to this workflow. This data is processed and manipulated within the cyber workflow. After processing, the results are returned to the physical component. The cyber workflow is formally represented as a directed acyclic graph (DAG), denoted as = ( , ). Here, is the set of tasks, and represents the dependencies or constraints between task pairs. Each task in this model has attributes like size and computational requirements.
Fog resources are distributed in each region, forming a network of interconnected nodes. These resources are linked together through wireless links, which can vary in characteristics from one resource to another. Furthermore, the resources are characterized by a set of attributes such as computational capacities. Our primary goal is to develop a decision-making algorithm that eficiently minimizes both the overall delay and energy consumption associated with processing in the system. This reduction includes ofloading processing, execution processing, and migration processing. It encompasses the time from when tasks are ofloaded to the resources of the initial region to the time the final task in the workflow completes its execution in the last region.
In contrast to other fields of Machine Learning, reinforcement learning relies on continuous interaction with the environment, where the agent learns through feedback in the form of values assessing its actions.
The workflow migration problem is formalized as a Markov Decision Process (MDP), denoted as = ⟨, , , ⟩ [13], where represents the state space, is the action space encompassing all possible actions at each state, is the reward function valuing state-action pairs, and determines the probability of transitioning between states when specific actions are taken. , , and are represented as follows: 1. State Space: The state space is defined by several variables that collectively represent the system state. These variables include the current time slot (), the information of the current task ( ) that needs to be allocated, the information of resources at the current region (), and the action taken for the task (− 1) in the previous time slot. The state can be denoted as = {, , , (−>11) }. The total number of states in one episode is equal to × , where represents the number of tasks, and denotes the number of time slots. 2. Action Space: In each time slot, an action must be taken for each task . The action ∈ {0(>1) , 1} consists of two options: 0 denotes no migration decision, while 1 involves migrating the task to the current region by selecting a suitable resource. 3. Reward: It represents delay and energy consumption associated with the execution of action within the context of state .
In RL, an agent interacts with an environment by taking actions, receiving rewards or penalties based on those actions, and using this feedback to improve its decision-making process. The agent’s actions are guided by a policy, which determines the mapping from system states to actions [13]. The ultimate objective of the agent is to ascertain an optimal policy, denoted as * , which efectively maps a state to a probability distribution over possible actions and is represented as follows:
* : → () = ∑︁ ∑︁ (, ) =1 =1 (1) (2)
In the initial stages, the agent operates without prior knowledge about the environment and therefore initiates exploration by taking random actions that might not yield immediate high rewards but ofer valuable insights for discovering more rewarding actions over time. Subsequently, the agent shifts to exploitation, where it selects actions aimed at maximizing the expected future rewards, relying on its current understanding of the environment. update is a fundamental aspect of RL algorithms like Q-learning and State-Action-Reward-StateAction (SARSA).
In the inference phase, the agent leverages its acquired knowledge to make decisions and take actions during interactions with its environment. This behavior reflects the wisdom accumulated from its training, guiding it to navigate and interact in alignment with the optimal migration policy * .
This position paper addresses Workflow Migration in the context of Fog-Cloud Computing, especially in scenarios with varying resource capacities and bandwidth links. We began by highlighting limitations and challenges in existing literature. We then outlined our system model. We introduced an MDP model, defining its key components. Additionally, we presented an RL framework for necessary workflow migration.
In the future, we aim to develop a Deep Learning resource prediction module using Google cluster trace and Alibaba data, explore partial ofloading for energy-constrained end-users, and investigate multi-agent strategies for scalability in expanding resource scenarios.
This work was partially supported by the LABEX-TA project MeFoGL: "Méthode Formelles pour le Génie Logiciel" [8] M. Zhang, H. Huang, L. Rui, G. Hui, Y. Wang, X. Qiu, A service migration method based on dynamic awareness in mobile edge computing, in: NOMS 2020-2020 IEEE/IFIP Network Operations and Management Symposium, IEEE, 2020, pp. 1–7. [9] F. De Vita, D. Bruneo, A. Puliafito, G. Nardini, A. Virdis, G. Stea, A deep reinforcement learning approach for data migration in multi-access edge computing, in: 2018 ITU Kaleidoscope: Machine Learning for a 5G Future (ITU K), IEEE, 2018, pp. 1–8. [10] T. Djemai, P. Stolf, T. Monteil, J.-M. Pierson, Mobility support for energy and qos aware iot services placement in the fog, in: 2020 International Conference on Software, Telecommunications and Computer Networks (SoftCOM), IEEE, 2020, pp. 1–7. [11] S.-Z. Huang, K.-Y. Lin, C.-L. Hu, Intelligent task migration with deep qlearning in multiaccess edge computing, IET Communications 16 (2022) 1290–1302. [12] P. Tian, G. Si, Z. An, J. Li, F. Zhou, Service migration strategy based on multi-attribute mdp in mobile edge computing, Electronics 11 (2022) 4070. [13] R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018.