Skip to content

lit_sb3.algorithms.her.her_replay_buffer

source module lit_sb3.algorithms.her.her_replay_buffer

Classes

source class HerReplayBuffer(buffer_size: int, observation_space: spaces.Dict, action_space: spaces.Space, env: VecEnv, device: torch.device | str = 'auto', n_envs: int = 1, optimize_memory_usage: bool = False, handle_timeout_termination: bool = True, n_sampled_goal: int = 4, goal_selection_strategy: GoalSelectionStrategy | str = 'future', copy_info_dict: bool = False)

Bases : DictReplayBuffer

Hindsight Experience Replay (HER) buffer.

Paper: https://arxiv.org/abs/1707.01495

Replay buffer for sampling HER (Hindsight Experience Replay) transitions.

.. note::

Compared to other implementations, the future goal sampling strategy is inclusive: the current transition can be used when re-sampling.

Parameters

  • buffer_size : int — Max number of element in the buffer

  • observation_space : spaces.Dict — Observation space

  • action_space : spaces.Space — Action space

  • env : VecEnv — The training environment

  • device : torch.device | str — PyTorch device

  • n_envs : int — Number of parallel environments

  • optimize_memory_usage : bool — Enable a memory efficient variant Disabled for now (see https://github.com/DLR-RM/stable-baselines3/pull/243#discussion_r531535702)

  • handle_timeout_termination : bool — Handle timeout termination (due to timelimit) separately and treat the task as infinite horizon task. https://github.com/DLR-RM/stable-baselines3/issues/284

  • n_sampled_goal : int — Number of virtual transitions to create per real transition, by sampling new goals.

  • goal_selection_strategy : GoalSelectionStrategy | str — Strategy for sampling goals for replay. One of ['episode', 'final', 'future']

  • copy_info_dict : bool — Whether to copy the info dictionary and pass it to compute_reward() method. Please note that the copy may cause a slowdown. False by default.

Methods

  • set_env — Sets the environment.

  • add

  • sample — Sample elements from the replay buffer.

  • truncate_last_trajectory — If called, we assume that the last trajectory in the replay buffer was finished.

source method HerReplayBuffer.set_env(env: VecEnv) → None

Sets the environment.

Parameters

Raises

  • ValueError

source method HerReplayBuffer.add(obs: dict[str, np.ndarray], next_obs: dict[str, np.ndarray], action: np.ndarray, reward: np.ndarray, done: np.ndarray, infos: list[dict[str, Any]]) → None

source method HerReplayBuffer.sample(batch_size: int, env: VecNormalize | None = None) → DictReplayBufferSamples

Sample elements from the replay buffer.

Parameters

  • batch_size : int — Number of element to sample

  • env : VecNormalize | None — Associated VecEnv to normalize the observations/rewards when sampling

Returns

Raises

  • RuntimeError

source method HerReplayBuffer.truncate_last_trajectory() → None

If called, we assume that the last trajectory in the replay buffer was finished.

(and truncate it). If not called, we assume that we continue the same trajectory (same episode).