Skip to content

lit_sb3.policies.actor_critic

source module lit_sb3.policies.actor_critic

Policies: abstract base class and concrete implementations.

Classes

  • ActorCriticPolicy — Policy class for actor-critic algorithms (has both policy and value prediction).

source class ActorCriticPolicy(observation_space: spaces.Space, action_space: spaces.Space, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.Tanh, ortho_init: bool = True, use_sde: bool = False, log_std_init: float = 0.0, full_std: bool = True, use_expln: bool = False, squash_output: bool = False, features_extractor_class: type[BaseFeaturesExtractor] = FlattenExtractor, features_extractor_kwargs: dict[str, Any] | None = None, share_features_extractor: bool = True, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : BasePolicy

Policy class for actor-critic algorithms (has both policy and value prediction).

Used by A2C, PPO and the likes.

Parameters

  • observation_space : spaces.Space — Observation space

  • action_space : spaces.Space — Action space

  • lr_schedule : Schedule — Learning rate schedule (could be constant)

  • net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.

  • activation_fn : type[nn.Module] — Activation function

  • ortho_init : bool — Whether to use or not orthogonal initialization

  • use_sde : bool — Whether to use State Dependent Exploration or not

  • log_std_init : float — Initial value for the log standard deviation

  • full_std : bool — Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE

  • use_expln : bool — Use expln() function instead of exp() to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.

  • squash_output : bool — Whether to squash the output using a tanh function, this allows to ensure boundaries when using gSDE.

  • features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.

  • features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.

  • share_features_extractor : bool — If True, the features extractor is shared between the policy and value networks.

  • normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

  • optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default

  • optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

  • device : torch.device — Infer which device this policy lives on by inspecting its parameters.

  • squash_output : bool — (bool) Getter for squash_output.

Methods

  • reset_noise — Sample new weights for the exploration matrix.

  • forward — Forward pass in all the networks (actor and critic).

  • extract_features — Preprocess the observation if needed and extract features.

  • evaluate_actions — Evaluate actions according to the current policy,.

  • get_distribution — Get the current policy distribution given the observations.

  • predict_values — Get the estimated values according to the current policy given the observations.

source method ActorCriticPolicy.reset_noise(n_envs: int = 1) → None

Sample new weights for the exploration matrix.

Parameters

  • n_envs : int

  • n_envs : int — The n envs parameter.

source method ActorCriticPolicy.forward(obs: torch.Tensor, deterministic: bool = False) → tuple[torch.Tensor, torch.Tensor, torch.Tensor]

Forward pass in all the networks (actor and critic).

Parameters

  • obs : torch.Tensor — Observation

  • deterministic : bool — Whether to sample or use deterministic actions

Returns

  • tuple[torch.Tensor, torch.Tensor, torch.Tensor] — action, value and log probability of the action

source method ActorCriticPolicy.extract_features(obs: PyTorchObs, features_extractor: BaseFeaturesExtractor | None = None) → torch.Tensor | tuple[torch.Tensor, torch.Tensor]

Preprocess the observation if needed and extract features.

Parameters

  • obs : PyTorchObs — Observation

  • features_extractor : BaseFeaturesExtractor | None — The features extractor to use. If None, then self.features_extractor is used.

Returns

  • torch.Tensor | tuple[torch.Tensor, torch.Tensor] — The extracted features. If features extractor is not shared, returns a tuple with the features for the actor and the features for the critic.

source method ActorCriticPolicy.evaluate_actions(obs: PyTorchObs, actions: torch.Tensor) → tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]

Evaluate actions according to the current policy,.

given the observations.

Parameters

  • obs : PyTorchObs — Observation

  • actions : torch.Tensor — Actions

Returns

  • tuple[torch.Tensor, torch.Tensor, torch.Tensor | None] — estimated value, log likelihood of taking those actions and entropy of the action distribution.

source method ActorCriticPolicy.get_distribution(obs: PyTorchObs) → Distribution

Get the current policy distribution given the observations.

Parameters

  • obs : PyTorchObs

  • obs : PyTorchObs — The current observation.

Returns

source method ActorCriticPolicy.predict_values(obs: PyTorchObs) → torch.Tensor

Get the estimated values according to the current policy given the observations.

Parameters

  • obs : PyTorchObs — Observation

Returns

  • torch.Tensor — the estimated values.