Reinforcement Learning (RL) is a strategic approach encompassing various algorithms, including Q-learning and Deep Q-learning Networks (DQN) [8], employed to optimize routing paths in dynamic IoT networks. Here’s a breakdown of the steps involved in implementing one of these algorithms.

Algorithm 1. Q learning.

1. Initialize Q-Table: Set up a Q-table with initial values for each state-action pair.

2. Set Parameters: Define the learning rate α, discount factor γ, and exploration rate ϵ.

3. Loop Until Convergence: Repeat the following steps for each episode.

4. Observe Current State: Begin in an initial state.

5. Choose Action: Select an action using either exploration (random choice) or exploitation (highest Q- value).

6. Take Action and Observe Reward: Execute the chosen action, move to a new state ′, and receive a reward r.

7. Update Q-Value: Update Q(s, a) using the Q-learning formula based on the reward and future Q-values.

8. Set New State: Update the current state to s′ for the next iteration.

9. Decay Exploration Rate (optional): Gradually reduce ϵ to increase exploitation over time.

10. Repeat: Continue looping until Q-values converge to stable values.

Fig. 3 shows the flow chart of the Q-learning algorithm.

RL is a strategic approach encompassing various algorithms, including Q-learning and DQN, employed to optimize routing paths in dynamic IoT networks. Here’s a breakdown of the steps involved in implementing one of these algorithms.

Algorithm 2. Deep Q-networks initialize replay memory D to capacity N.

Initialize action-value function Q with random weights for episode = 1, M do Initialise sequence s1 = {x1} and preprocessed sequenced φ1 = φ(s1) for t = 1, T do

With probability e, select a random action ot;otherwise, select at = max Q∗(φ(st), a; θ) Execute action in the emulator and observe reward rt and image xt+1 Set st+1 = st, at, xt+1 and preprocess φt+1 = φ(st+1) Store transition (φt, at, rt, φt+1) in D. Sample random minibatch of transitions (φj , aj , rj , φj+1) from D Set yj = { rj for terminal φj+1 rj + γ maxa′ Q(φj+1, a′; θ) for non-terminal φj+1 Perform a gradient descent step on (yj − Q(φj , aj ; θ))2 according to equation 3 end for end for

The SARSA algorithm is an on-policy RL approach for training agents to make sequential decisions [9]. Unlike Q-learning, which learns from actions outside the agent's current policy, SARSA updates its values based on its actions, leading to safer learning in uncertain environments. The agent selects an action, observes the resulting reward and next state, then chooses the following action based on its policy and updates its value estimates. This makes SARSA especially suited for situations where safe exploration and adapting to real-time feedback are essential.

SARSA is used in scenarios like autonomous navigation, gameplay, and agent training in dynamic environments where safety or penalties need to be considered carefully.

I. Initialize: Initialize Q-values for all state-action pairs (often randomly).

II. Select action (A): Choose an action based on the current state S, using a policy derived from Q-values, such as the ε-greedy policy.

III. Take action and observe reward: Take action A, observe the reward R, and the new state S′. Choose next action A′: Based on the new state S′, select the following action A′ using the same policy. Update Q-value: Update the Q-value for the state-action pair (S, A) using the following SARSA update rule:

Q(S,A)←Q(S,A)+α(R+γQ(S′,A′)−Q(S,A)) IV. α: Learning rate controls how much new information overrides old information.

V. γ: Discount factor represents the importance of future rewards.

VI. Repeat: Set S←S′ and A←A′, and repeat steps 3 to 5 until the episode ends.

Q-learning struggles with large state-action spaces, as significant memory and computational resources are required to store and update the Q-values for every possible state-action pair.

Q-learning requires discrete states, making it challenging to use directly in environments with continuous states without additional modifications like function approximations.

Q-Learning assigns a unique Q-value to each specific state-action pair, treating them as separate experiences. This lack of generalization means that similar states are not recognized as related, preventing the agent from applying knowledge gained in one area to another. As a result, the agent often re-learns similar information multiple times, leading to redundant computations and slower convergence, especially in environments with numerous or structurally similar states.

DQNs require large amounts of training data to learn effectively, as neural networks need numerous samples to approximate Q-values accurately, making training slow and resource-intensive.

DQNs can be unstable and struggle to converge, especially in environments with sparse rewards or delayed rewards. To improve stability, techniques like experience replay and target networks are often required.

DQNs can overfit to specific states seen during training, which limits their ability to generalize well in more complex or dynamic environments and reduces performance outside of training conditions.

SARSA’s reliance on observed rewards can lead to significant fluctuations in performance, particularly in environments with high variability in reward structures. High reward variance may cause SARSA to converge slowly or to a suboptimal policy due to the inconsistent reward feedback loop, leading to prolonged training times and unpredictable performance in real-world applications. This phenomenon is similar to the “credit assignment problem" in RL, where the agent struggles to identify actions that contribute to delayed rewards correctly.

SARSA's on-policy nature means it evaluates and improves the policy it uses to make decisions, relying heavily on a balanced exploration strategy. If the exploration policy is not well-designed, SARSA may excessively explore suboptimal regions of the state-action space, potentially resulting in slower convergence or convergence to a suboptimal solution. This issue, akin to “policy oscillation,” occurs when the agent repeatedly visits states without improving its understanding of optimal actions.

SARSA is sensitive to environmental changes since it updates the policy based on observed states and actions. This can lead to instability in highly dynamic environments, as SARSA may struggle to adapt quickly enough to changing conditions. This limitation is similar to the “catastrophic forgetting” problem, where an algorithm loses information about previously learned patterns due to new, conflicting information, often resulting in degraded performance over time.