lit_sb3.buffers.dict_rollout
source module lit_sb3.buffers.dict_rollout
Classes
-
DictRolloutBuffer — Dict Rollout buffer used in on-policy algorithms like A2C/PPO.
source class DictRolloutBuffer(buffer_size: int, observation_space: spaces.Dict, action_space: spaces.Space, device: torch.device | str = 'auto', gae_lambda: float = 1, gamma: float = 0.99, n_envs: int = 1)
Bases : RolloutBuffer
Dict Rollout buffer used in on-policy algorithms like A2C/PPO.
Extends the RolloutBuffer to use dictionary observations.
It corresponds to buffer_size transitions collected
using the current policy.
This experience will be discarded after the policy update.
In order to use PPO objective, we also store the current value of each state
and the log probability of each taken action.
The term rollout here refers to the model-free notion and should not be used with the concept of rollout used in model-based RL or planning. Hence, it is only involved in policy and value function training but not action selection.
Parameters
-
buffer_size : int — Max number of element in the buffer
-
observation_space : spaces.Dict — Observation space
-
action_space : spaces.Space — Action space
-
device : torch.device | str — PyTorch device
-
gae_lambda : float — Factor for trade-off of bias vs variance for Generalized Advantage Estimator Equivalent to Monte-Carlo advantage estimate when set to 1.
-
gamma : float — Discount factor
-
n_envs : int — Number of parallel environments
Methods
source method DictRolloutBuffer.reset() → None
source method DictRolloutBuffer.add(obs: dict[str, np.ndarray], action: np.ndarray, reward: np.ndarray, episode_start: np.ndarray, value: torch.Tensor, log_prob: torch.Tensor) → None
Args.
obs: Observation action: Action reward: episode_start: Start of episode signal. value: estimated value of the current state following the current policy. log_prob: log probability of the action following the current policy.
source method DictRolloutBuffer.get(batch_size: int | None = None) → Generator[DictRolloutBufferSamples, None, None]