Skip to content

lit_sb3.buffers.nstep_replay

source module lit_sb3.buffers.nstep_replay

Classes

  • NStepReplayBuffer — Replay buffer used for computing n-step returns in off-policy algorithms like SAC/DQN.

source class NStepReplayBuffer(*args, n_steps: int = 3, gamma: float = 0.99, **kwargs)

Bases : ReplayBuffer

Replay buffer used for computing n-step returns in off-policy algorithms like SAC/DQN.

The n-step return combines multiple steps of future rewards, discounted by the discount factor gamma. This can help improve sample efficiency and credit assignment.

This implementation uses the same storage space as a normal replay buffer, and NumPy vectorized operations at sampling time to efficiently compute the n-step return, without requiring extra memory.

This implementation is inspired by

  • https://github.com/younggyoseo/FastTD3
  • https://github.com/DLR-RM/stable-baselines3/pull/81

It avoids potential issues such as

  • https://github.com/younggyoseo/FastTD3/issues/6

Parameters

  • buffer_size — Max number of element in the buffer

  • observation_space — Observation space

  • action_space — Action space

  • device — PyTorch device

  • n_envs — Number of parallel environments

  • optimize_memory_usage — Not supported

  • handle_timeout_termination — Handle timeout termination (due to timelimit) separately and treat the task as infinite horizon task. https://github.com/DLR-RM/stable-baselines3/issues/284

  • n_steps : int — Number of steps to accumulate rewards for n-step returns

  • gamma : float — Discount factor for future rewards