lit_sb3.policies.recurrent.recurrent_actor_critic

source module lit_sb3.policies.recurrent.recurrent_actor_critic

Classes

• RecurrentActorCriticPolicy — Recurrent policy class for actor-critic algorithms (has both policy and value prediction).

source class RecurrentActorCriticPolicy(observation_space: spaces.Space, action_space: spaces.Space, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.Tanh, ortho_init: bool = True, use_sde: bool = False, log_std_init: float = 0.0, full_std: bool = True, use_expln: bool = False, squash_output: bool = False, features_extractor_class: type[BaseFeaturesExtractor] = FlattenExtractor, features_extractor_kwargs: dict[str, Any] | None = None, share_features_extractor: bool = True, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None, lstm_hidden_size: int = 256, n_lstm_layers: int = 1, shared_lstm: bool = False, enable_critic_lstm: bool = True, lstm_kwargs: dict[str, Any] | None = None)

Bases : ActorCriticPolicy

Recurrent policy class for actor-critic algorithms (has both policy and value prediction).

To be used with A2C, PPO and the likes. It assumes that both the actor and the critic LSTM have the same architecture.

Parameters

• observation_space : spaces.Space — Observation space

• action_space : spaces.Space — Action space

• lr_schedule : Schedule — Learning rate schedule (could be constant)

• net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.

• activation_fn : type[nn.Module] — Activation function

• ortho_init : bool — Whether to use or not orthogonal initialization

• use_sde : bool — Whether to use State Dependent Exploration or not

• log_std_init : float — Initial value for the log standard deviation

• full_std : bool — Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE

• use_expln : bool — Use expln() function instead of exp() to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.

• squash_output : bool — Whether to squash the output using a tanh function, this allows to ensure boundaries when using gSDE.

• features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.

• features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.

• share_features_extractor : bool — If True, the features extractor is shared between the policy and value networks.

• normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

• optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default

• optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

• lstm_hidden_size : int — Number of hidden units for each LSTM layer.

• n_lstm_layers : int — Number of LSTM layers.

• shared_lstm : bool — Whether the LSTM is shared between the actor and the critic (in that case, only the actor gradient is used) By default, the actor and the critic have two separate LSTM.

• enable_critic_lstm : bool — Use a seperate LSTM for the critic.

• lstm_kwargs : dict[str, Any] | None — Additional keyword arguments to pass the the LSTM constructor.

Attributes

• device : torch.device — Infer which device this policy lives on by inspecting its parameters.

• squash_output : bool — (bool) Getter for squash_output.

Methods

• forward — Forward pass in all the networks (actor and critic).

• get_distribution — Get the current policy distribution given the observations.

• predict_values — Get the estimated values according to the current policy given the observations.

• evaluate_actions — Evaluate actions according to the current policy,.

• predict — Get the policy action from an observation (and optional hidden state).

source method RecurrentActorCriticPolicy.forward(obs: torch.Tensor, lstm_states: RNNStates, episode_starts: torch.Tensor, deterministic: bool = False) → tuple[torch.Tensor, torch.Tensor, torch.Tensor, RNNStates]

Forward pass in all the networks (actor and critic).

Parameters

• obs : torch.Tensor — Observation. Observation

• lstm_states : RNNStates — The last hidden and memory states for the LSTM.

• episode_starts : torch.Tensor — Whether the observations correspond to new episodes or not (we reset the lstm states in that case).

• deterministic : bool — Whether to sample or use deterministic actions

Returns

• tuple[torch.Tensor, torch.Tensor, torch.Tensor, RNNStates] — action, value and log probability of the action

source method RecurrentActorCriticPolicy.get_distribution(obs: torch.Tensor, lstm_states: tuple[torch.Tensor, torch.Tensor], episode_starts: torch.Tensor) → tuple[Distribution, tuple[torch.Tensor, ...]]

Get the current policy distribution given the observations.

Parameters

• obs : torch.Tensor — Observation.

• lstm_states : tuple[torch.Tensor, torch.Tensor] — The last hidden and memory states for the LSTM.

• episode_starts : torch.Tensor — Whether the observations correspond to new episodes or not (we reset the lstm states in that case).

Returns

• tuple[Distribution, tuple[torch.Tensor, ...]] — the action distribution and new hidden states.

source method RecurrentActorCriticPolicy.predict_values(obs: torch.Tensor, lstm_states: tuple[torch.Tensor, torch.Tensor], episode_starts: torch.Tensor) → torch.Tensor

Get the estimated values according to the current policy given the observations.

Parameters

• obs : torch.Tensor — Observation.

• lstm_states : tuple[torch.Tensor, torch.Tensor] — The last hidden and memory states for the LSTM.

• episode_starts : torch.Tensor — Whether the observations correspond to new episodes or not (we reset the lstm states in that case).

Returns

• torch.Tensor — the estimated values.

source method RecurrentActorCriticPolicy.evaluate_actions(obs: torch.Tensor, actions: torch.Tensor, lstm_states: RNNStates, episode_starts: torch.Tensor) → tuple[torch.Tensor, torch.Tensor, torch.Tensor]

Evaluate actions according to the current policy,.

given the observations.

Parameters

• obs : torch.Tensor — Observation.

• actions : torch.Tensor

• lstm_states : RNNStates — The last hidden and memory states for the LSTM.

• episode_starts : torch.Tensor — Whether the observations correspond to new episodes or not (we reset the lstm states in that case).

• actions : torch.Tensor — Actions taken.

Returns

• tuple[torch.Tensor, torch.Tensor, torch.Tensor] — estimated value, log likelihood of taking those actions and entropy of the action distribution.

source method RecurrentActorCriticPolicy.predict(observation: np.ndarray | dict[str, np.ndarray], state: tuple[np.ndarray, ...] | None = None, episode_start: np.ndarray | None = None, deterministic: bool = False) → tuple[np.ndarray, tuple[np.ndarray, ...] | None]

Get the policy action from an observation (and optional hidden state).

Includes sugar-coating to handle different observations (e.g. normalizing images).

Parameters

• observation : np.ndarray | dict[str, np.ndarray] — the input observation

• lstm_states — The last hidden and memory states for the LSTM.

• episode_starts — Whether the observations correspond to new episodes or not (we reset the lstm states in that case).

• deterministic : bool — Whether or not to return deterministic actions.

• state : tuple[np.ndarray, ...] | None — The last hidden states for the LSTM.

• episode_start : np.ndarray | None — Whether the episode is starting (triggers lstm state reset).

Returns

• tuple[np.ndarray, tuple[np.ndarray, ...] | None] — the model's action and the next hidden state (used in recurrent policies)