lit_sb3.policies.maskable.actor_critic

source module lit_sb3.policies.maskable.actor_critic

Classes

• MaskableActorCriticPolicy — Policy class for actor-critic algorithms (has both policy and value prediction).

source class MaskableActorCriticPolicy(observation_space: spaces.Space, action_space: spaces.Space, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.Tanh, ortho_init: bool = True, features_extractor_class: type[BaseFeaturesExtractor] = FlattenExtractor, features_extractor_kwargs: dict[str, Any] | None = None, share_features_extractor: bool = True, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : BasePolicy

Policy class for actor-critic algorithms (has both policy and value prediction).

Used by A2C, PPO and the likes.

Parameters

• observation_space : spaces.Space — Observation space

• action_space : spaces.Space — Action space

• lr_schedule : Schedule — Learning rate schedule (could be constant)

• net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.

• activation_fn : type[nn.Module] — Activation function

• ortho_init : bool — Whether to use or not orthogonal initialization

• features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.

• features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.

• share_features_extractor : bool — If True, the features extractor is shared between the policy and value networks.

• normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

• optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default

• optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

• device : torch.device — Infer which device this policy lives on by inspecting its parameters.

• squash_output : bool — (bool) Getter for squash_output.

Methods

• forward — Forward pass in all the networks (actor and critic).

• extract_features — Preprocess the observation if needed and extract features.

• predict — Get the policy action from an observation (and optional hidden state).

• evaluate_actions — Evaluate actions according to the current policy,.

• get_distribution — Get the current policy distribution given the observations.

• predict_values — Get the estimated values according to the current policy given the observations.

source method MaskableActorCriticPolicy.forward(obs: torch.Tensor, deterministic: bool = False, action_masks: np.ndarray | None = None) → tuple[torch.Tensor, torch.Tensor, torch.Tensor]

Forward pass in all the networks (actor and critic).

Parameters

• obs : torch.Tensor — Observation

• deterministic : bool — Whether to sample or use deterministic actions

• action_masks : np.ndarray | None — Action masks to apply to the action distribution

Returns

• tuple[torch.Tensor, torch.Tensor, torch.Tensor] — action, value and log probability of the action

source method MaskableActorCriticPolicy.extract_features(obs: PyTorchObs, features_extractor: BaseFeaturesExtractor | None = None) → torch.Tensor | tuple[torch.Tensor, torch.Tensor]

Preprocess the observation if needed and extract features.

Parameters

• obs : PyTorchObs — Observation

• features_extractor : BaseFeaturesExtractor | None — The features extractor to use. If None, then self.features_extractor is used.

Returns

• torch.Tensor | tuple[torch.Tensor, torch.Tensor] — The extracted features. If features extractor is not shared, returns a tuple with the features for the actor and the features for the critic.

source method MaskableActorCriticPolicy.predict(observation: np.ndarray | dict[str, np.ndarray], state: tuple[np.ndarray, ...] | None = None, episode_start: np.ndarray | None = None, deterministic: bool = False, action_masks: np.ndarray | None = None) → tuple[np.ndarray, tuple[np.ndarray, ...] | None]

Get the policy action from an observation (and optional hidden state).

Includes sugar-coating to handle different observations (e.g. normalizing images).

Parameters

• observation : np.ndarray | dict[str, np.ndarray] — the input observation

• state : tuple[np.ndarray, ...] | None — The last states (can be None, used in recurrent policies)

• episode_start : np.ndarray | None — The last masks (can be None, used in recurrent policies)

• deterministic : bool — Whether or not to return deterministic actions.

• action_masks : np.ndarray | None — Action masks to apply to the action distribution

Returns

• tuple[np.ndarray, tuple[np.ndarray, ...] | None] — the model's action and the next state (used in recurrent policies)

Raises

• ValueError

source method MaskableActorCriticPolicy.evaluate_actions(obs: torch.Tensor, actions: torch.Tensor, action_masks: torch.Tensor | None = None) → tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]

Evaluate actions according to the current policy,.

given the observations.

Parameters

• obs : torch.Tensor — Observation

• actions : torch.Tensor — Actions

• action_masks : torch.Tensor | None — Boolean masks for invalid actions (True = valid).

Returns

• tuple[torch.Tensor, torch.Tensor, torch.Tensor | None] — estimated value, log likelihood of taking those actions and entropy of the action distribution.

source method MaskableActorCriticPolicy.get_distribution(obs: PyTorchObs, action_masks: np.ndarray | None = None) → MaskableDistribution

Get the current policy distribution given the observations.

Parameters

• obs : PyTorchObs — Observation

• action_masks : np.ndarray | None — Actions' mask

Returns

• MaskableDistribution — the action distribution.

source method MaskableActorCriticPolicy.predict_values(obs: PyTorchObs) → torch.Tensor

Get the estimated values according to the current policy given the observations.

Parameters

• obs : PyTorchObs — Observation

Returns

• torch.Tensor — the estimated values.