lit_sb3.policies

Classes

ActorCriticCnnPolicy — CNN policy class for actor-critic algorithms (has both policy and value prediction).
ActorCriticPolicy — Policy class for actor-critic algorithms (has both policy and value prediction).
BaseModel — The base model object: makes predictions in response to observations.
BasePolicy — The base policy object.
ContinuousCritic — Critic network(s) for DDPG/SAC/TD3.
MaskableActorCriticCnnPolicy — CNN policy class for actor-critic algorithms (has both policy and value prediction).
MaskableActorCriticPolicy — Policy class for actor-critic algorithms (has both policy and value prediction).
MaskableMultiInputActorCriticPolicy — MultiInputActorClass policy class for actor-critic algorithms (has both policy and value prediction).
MultiInputActorCriticPolicy — MultiInputActorClass policy class for actor-critic algorithms (has both policy and value prediction).

source class ActorCriticCnnPolicy(observation_space: spaces.Space, action_space: spaces.Space, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.Tanh, ortho_init: bool = True, use_sde: bool = False, log_std_init: float = 0.0, full_std: bool = True, use_expln: bool = False, squash_output: bool = False, features_extractor_class: type[BaseFeaturesExtractor] = NatureCNN, features_extractor_kwargs: dict[str, Any] | None = None, share_features_extractor: bool = True, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : ActorCriticPolicy

CNN policy class for actor-critic algorithms (has both policy and value prediction).

Used by A2C, PPO and the likes.

Parameters

observation_space : spaces.Space — Observation space
action_space : spaces.Space — Action space
lr_schedule : Schedule — Learning rate schedule (could be constant)
net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.
activation_fn : type[nn.Module] — Activation function
ortho_init : bool — Whether to use or not orthogonal initialization
use_sde : bool — Whether to use State Dependent Exploration or not
log_std_init : float — Initial value for the log standard deviation
full_std : bool — Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE
use_expln : bool — Use expln() function instead of exp() to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.
squash_output : bool — Whether to squash the output using a tanh function, this allows to ensure boundaries when using gSDE.
features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.
features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.
share_features_extractor : bool — If True, the features extractor is shared between the policy and value networks.
normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)
optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default
optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.
squash_output : bool — (bool) Getter for squash_output.

source class ActorCriticPolicy(observation_space: spaces.Space, action_space: spaces.Space, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.Tanh, ortho_init: bool = True, use_sde: bool = False, log_std_init: float = 0.0, full_std: bool = True, use_expln: bool = False, squash_output: bool = False, features_extractor_class: type[BaseFeaturesExtractor] = FlattenExtractor, features_extractor_kwargs: dict[str, Any] | None = None, share_features_extractor: bool = True, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : BasePolicy

Policy class for actor-critic algorithms (has both policy and value prediction).

Used by A2C, PPO and the likes.

Parameters

observation_space : spaces.Space — Observation space
action_space : spaces.Space — Action space
lr_schedule : Schedule — Learning rate schedule (could be constant)
net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.
activation_fn : type[nn.Module] — Activation function
ortho_init : bool — Whether to use or not orthogonal initialization
use_sde : bool — Whether to use State Dependent Exploration or not
log_std_init : float — Initial value for the log standard deviation
full_std : bool — Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE
use_expln : bool — Use expln() function instead of exp() to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.
squash_output : bool — Whether to squash the output using a tanh function, this allows to ensure boundaries when using gSDE.
features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.
features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.
share_features_extractor : bool — If True, the features extractor is shared between the policy and value networks.
normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)
optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default
optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.
squash_output : bool — (bool) Getter for squash_output.

Methods

reset_noise — Sample new weights for the exploration matrix.
forward — Forward pass in all the networks (actor and critic).
extract_features — Preprocess the observation if needed and extract features.
evaluate_actions — Evaluate actions according to the current policy,.
get_distribution — Get the current policy distribution given the observations.
predict_values — Get the estimated values according to the current policy given the observations.

source method ActorCriticPolicy.reset_noise(n_envs: int = 1) → None

Sample new weights for the exploration matrix.

Parameters

n_envs : int
n_envs : int — The n envs parameter.

source method ActorCriticPolicy.forward(obs: torch.Tensor, deterministic: bool = False) → tuple[torch.Tensor, torch.Tensor, torch.Tensor]

Forward pass in all the networks (actor and critic).

Parameters

obs : torch.Tensor — Observation
deterministic : bool — Whether to sample or use deterministic actions

Returns

tuple[torch.Tensor, torch.Tensor, torch.Tensor] — action, value and log probability of the action

source method ActorCriticPolicy.extract_features(obs: PyTorchObs, features_extractor: BaseFeaturesExtractor | None = None) → torch.Tensor | tuple[torch.Tensor, torch.Tensor]

Preprocess the observation if needed and extract features.

Parameters

obs : PyTorchObs — Observation
features_extractor : BaseFeaturesExtractor | None — The features extractor to use. If None, then self.features_extractor is used.

Returns

torch.Tensor | tuple[torch.Tensor, torch.Tensor] — The extracted features. If features extractor is not shared, returns a tuple with the features for the actor and the features for the critic.

source method ActorCriticPolicy.evaluate_actions(obs: PyTorchObs, actions: torch.Tensor) → tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]

Evaluate actions according to the current policy,.

given the observations.

Parameters

obs : PyTorchObs — Observation
actions : torch.Tensor — Actions

Returns

tuple[torch.Tensor, torch.Tensor, torch.Tensor | None] — estimated value, log likelihood of taking those actions and entropy of the action distribution.

source method ActorCriticPolicy.get_distribution(obs: PyTorchObs) → Distribution

Get the current policy distribution given the observations.

Parameters

obs : PyTorchObs
obs : PyTorchObs — The current observation.

Returns

Distribution — the action distribution.

source method ActorCriticPolicy.predict_values(obs: PyTorchObs) → torch.Tensor

Get the estimated values according to the current policy given the observations.

Parameters

obs : PyTorchObs — Observation

Returns

torch.Tensor — the estimated values.

source class BaseModel(observation_space: spaces.Space, action_space: spaces.Space, features_extractor_class: type[BaseFeaturesExtractor] = FlattenExtractor, features_extractor_kwargs: dict[str, Any] | None = None, features_extractor: BaseFeaturesExtractor | None = None, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : nn.Module

The base model object: makes predictions in response to observations.

In the case of policies, the prediction is an action. In the case of critics, it is the estimated value of the observation.

Parameters

observation_space : spaces.Space — The observation space of the environment
action_space : spaces.Space — The action space of the environment
features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.
features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.
features_extractor : BaseFeaturesExtractor | None — Network to extract features (a CNN when using images, a nn.Flatten() layer otherwise)
normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)
optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default
optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.

Methods

make_features_extractor — Helper method to create a features extractor.
extract_features — Preprocess the observation if needed and extract features.
save — Save model to a given location.
load — Load model from path.
load_from_vector — Load parameters from a 1D vector.
parameters_to_vector — Convert the parameters to a 1D vector.
set_training_mode — Put the policy in either training or evaluation mode.
is_vectorized_observation — Check whether or not the observation is vectorized,.
obs_to_tensor — Convert an input observation to a PyTorch tensor that can be fed to a model.

source method BaseModel.make_features_extractor() → BaseFeaturesExtractor

Helper method to create a features extractor.

source method BaseModel.extract_features(obs: PyTorchObs, features_extractor: BaseFeaturesExtractor) → torch.Tensor

Preprocess the observation if needed and extract features.

Parameters

obs : PyTorchObs — Observation
features_extractor : BaseFeaturesExtractor — The features extractor to use.

Returns

torch.Tensor — The extracted features

source property BaseModel.device: torch.device

Infer which device this policy lives on by inspecting its parameters.

If it has no parameters, the 'cpu' device is used as a fallback.

source method BaseModel.save(path: str) → None

Save model to a given location.

Parameters

path : str
path : str — Path to save the policy to.

source classmethod BaseModel.load(path: str, device: torch.device | str = 'auto') → SelfBaseModel

Load model from path.

Parameters

path : str
device : torch.device | str — Device on which the policy should be loaded.
path : str — Path to save the policy to.

source method BaseModel.load_from_vector(vector: np.ndarray) → None

Load parameters from a 1D vector.

Parameters

vector : np.ndarray
vector : np.ndarray — The vector parameter.

source method BaseModel.parameters_to_vector() → np.ndarray

Convert the parameters to a 1D vector.

source method BaseModel.set_training_mode(mode: bool) → None

Put the policy in either training or evaluation mode.

This affects certain modules, such as batch normalisation and dropout.

Parameters

mode : bool — if true, set to training mode, else set to evaluation mode

source method BaseModel.is_vectorized_observation(observation: np.ndarray | dict[str, np.ndarray]) → bool

Check whether or not the observation is vectorized,.

apply transposition to image (so that they are channel-first) if needed. This is used in DQN when sampling random action (epsilon-greedy policy).

Parameters

observation : np.ndarray | dict[str, np.ndarray] — the input observation to check

Returns

bool — whether the given observation is vectorized or not

source method BaseModel.obs_to_tensor(observation: np.ndarray | dict[str, np.ndarray]) → tuple[PyTorchObs, bool]

Convert an input observation to a PyTorch tensor that can be fed to a model.

Includes sugar-coating to handle different observations (e.g. normalizing images).

Parameters

observation : np.ndarray | dict[str, np.ndarray] — the input observation

Returns

tuple[PyTorchObs, bool] — The observation as PyTorch tensor and whether the observation is vectorized or not

source class BasePolicy(*args, squash_output: bool = False, **kwargs)

Bases : BaseModel, ABC

The base policy object.

Parameters are mostly the same as BaseModel; additions are documented below.

Parameters

args — positional arguments passed through to BaseModel.
kwargs — keyword arguments passed through to BaseModel.
squash_output : bool — For continuous actions, whether the output is squashed or not using a tanh() function.

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.
squash_output : bool — (bool) Getter for squash_output.

Methods

init_weights — Orthogonal initialization (used in PPO and A2C).
predict — Get the policy action from an observation (and optional hidden state).
scale_action — Rescale the action from [low, high] to [-1, 1].
unscale_action — Rescale the action from [-1, 1] to [low, high].

source property BasePolicy.squash_output: bool

(bool) Getter for squash_output.

source staticmethod BasePolicy.init_weights(module: nn.Module, gain: float = 1) → None

Orthogonal initialization (used in PPO and A2C).

source method BasePolicy.predict(observation: np.ndarray | dict[str, np.ndarray], state: tuple[np.ndarray, ...] | None = None, episode_start: np.ndarray | None = None, deterministic: bool = False) → tuple[np.ndarray, tuple[np.ndarray, ...] | None]

Get the policy action from an observation (and optional hidden state).

Includes sugar-coating to handle different observations (e.g. normalizing images).

Parameters

observation : np.ndarray | dict[str, np.ndarray] — the input observation
state : tuple[np.ndarray, ...] | None — The last hidden states (can be None, used in recurrent policies)
episode_start : np.ndarray | None — The last masks (can be None, used in recurrent policies) this correspond to beginning of episodes, where the hidden states of the RNN must be reset.
deterministic : bool — Whether or not to return deterministic actions.

Returns

tuple[np.ndarray, tuple[np.ndarray, ...] | None] — the model's action and the next hidden state (used in recurrent policies)

Raises

ValueError

source method BasePolicy.scale_action(action: np.ndarray) → np.ndarray

Rescale the action from [low, high] to [-1, 1].

(no need for symmetric action space).

Parameters

action : np.ndarray — Action to scale

Returns

np.ndarray — Scaled action

source method BasePolicy.unscale_action(scaled_action: np.ndarray) → np.ndarray

Rescale the action from [-1, 1] to [low, high].

(no need for symmetric action space).

Parameters

scaled_action : np.ndarray — Action to un-scale

source class ContinuousCritic(observation_space: spaces.Space, action_space: spaces.Box, net_arch: list[int], features_extractor: BaseFeaturesExtractor, features_dim: int, activation_fn: type[nn.Module] = nn.ReLU, normalize_images: bool = True, n_critics: int = 2, share_features_extractor: bool = True)

Bases : BaseModel

Critic network(s) for DDPG/SAC/TD3.

It represents the action-state value function (Q-value function). Compared to A2C/PPO critics, this one represents the Q-value and takes the continuous action as input. It is concatenated with the state and then fed to the network which outputs a single value: Q(s, a). For more recent algorithms like SAC/TD3, multiple networks are created to give different estimates.

By default, it creates two critic networks used to reduce overestimation thanks to clipped Q-learning (cf TD3 paper).

Parameters

observation_space : spaces.Space — Observation space
action_space : spaces.Box — Action space
net_arch : list[int] — Network architecture
features_extractor : BaseFeaturesExtractor — Network to extract features (a CNN when using images, a nn.Flatten() layer otherwise)
features_dim : int — Number of features
activation_fn : type[nn.Module] — Activation function
normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)
n_critics : int — Number of critic networks to create.
share_features_extractor : bool — Whether the features extractor is shared or not between the actor and the critic (this saves computation time)

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.

Methods

forward
q1_forward — Only predict the Q-value using the first network.

source method ContinuousCritic.forward(obs: torch.Tensor, actions: torch.Tensor) → tuple[torch.Tensor, ...]

source method ContinuousCritic.q1_forward(obs: torch.Tensor, actions: torch.Tensor) → torch.Tensor

Only predict the Q-value using the first network.

This allows to reduce computation when all the estimates are not needed (e.g. when updating the policy in TD3).

source class MaskableActorCriticCnnPolicy(observation_space: spaces.Space, action_space: spaces.Space, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.Tanh, ortho_init: bool = True, features_extractor_class: type[BaseFeaturesExtractor] = NatureCNN, features_extractor_kwargs: dict[str, Any] | None = None, share_features_extractor: bool = True, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : MaskableActorCriticPolicy

CNN policy class for actor-critic algorithms (has both policy and value prediction).

Used by A2C, PPO and the likes.

Parameters

observation_space : spaces.Space — Observation space
action_space : spaces.Space — Action space
lr_schedule : Schedule — Learning rate schedule (could be constant)
net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.
activation_fn : type[nn.Module] — Activation function
ortho_init : bool — Whether to use or not orthogonal initialization
features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.
features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.
share_features_extractor : bool — If True, the features extractor is shared between the policy and value networks.
normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)
optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default
optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.
squash_output : bool — (bool) Getter for squash_output.

source class MaskableActorCriticPolicy(observation_space: spaces.Space, action_space: spaces.Space, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.Tanh, ortho_init: bool = True, features_extractor_class: type[BaseFeaturesExtractor] = FlattenExtractor, features_extractor_kwargs: dict[str, Any] | None = None, share_features_extractor: bool = True, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : BasePolicy

Policy class for actor-critic algorithms (has both policy and value prediction).

Used by A2C, PPO and the likes.

Parameters

observation_space : spaces.Space — Observation space
action_space : spaces.Space — Action space
lr_schedule : Schedule — Learning rate schedule (could be constant)
net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.
activation_fn : type[nn.Module] — Activation function
ortho_init : bool — Whether to use or not orthogonal initialization
features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.
features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.
share_features_extractor : bool — If True, the features extractor is shared between the policy and value networks.
normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)
optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default
optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.
squash_output : bool — (bool) Getter for squash_output.

Methods

forward — Forward pass in all the networks (actor and critic).
extract_features — Preprocess the observation if needed and extract features.
predict — Get the policy action from an observation (and optional hidden state).
evaluate_actions — Evaluate actions according to the current policy,.
get_distribution — Get the current policy distribution given the observations.
predict_values — Get the estimated values according to the current policy given the observations.

source method MaskableActorCriticPolicy.forward(obs: torch.Tensor, deterministic: bool = False, action_masks: np.ndarray | None = None) → tuple[torch.Tensor, torch.Tensor, torch.Tensor]

Forward pass in all the networks (actor and critic).

Parameters

obs : torch.Tensor — Observation
deterministic : bool — Whether to sample or use deterministic actions
action_masks : np.ndarray | None — Action masks to apply to the action distribution

Returns

tuple[torch.Tensor, torch.Tensor, torch.Tensor] — action, value and log probability of the action

source method MaskableActorCriticPolicy.extract_features(obs: PyTorchObs, features_extractor: BaseFeaturesExtractor | None = None) → torch.Tensor | tuple[torch.Tensor, torch.Tensor]

Preprocess the observation if needed and extract features.

Parameters

obs : PyTorchObs — Observation
features_extractor : BaseFeaturesExtractor | None — The features extractor to use. If None, then self.features_extractor is used.

Returns

torch.Tensor | tuple[torch.Tensor, torch.Tensor] — The extracted features. If features extractor is not shared, returns a tuple with the features for the actor and the features for the critic.

source method MaskableActorCriticPolicy.predict(observation: np.ndarray | dict[str, np.ndarray], state: tuple[np.ndarray, ...] | None = None, episode_start: np.ndarray | None = None, deterministic: bool = False, action_masks: np.ndarray | None = None) → tuple[np.ndarray, tuple[np.ndarray, ...] | None]

Get the policy action from an observation (and optional hidden state).

Includes sugar-coating to handle different observations (e.g. normalizing images).

Parameters

observation : np.ndarray | dict[str, np.ndarray] — the input observation
state : tuple[np.ndarray, ...] | None — The last states (can be None, used in recurrent policies)
episode_start : np.ndarray | None — The last masks (can be None, used in recurrent policies)
deterministic : bool — Whether or not to return deterministic actions.
action_masks : np.ndarray | None — Action masks to apply to the action distribution

Returns

tuple[np.ndarray, tuple[np.ndarray, ...] | None] — the model's action and the next state (used in recurrent policies)

Raises

ValueError

source method MaskableActorCriticPolicy.evaluate_actions(obs: torch.Tensor, actions: torch.Tensor, action_masks: torch.Tensor | None = None) → tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]

Evaluate actions according to the current policy,.

given the observations.

Parameters

obs : torch.Tensor — Observation
actions : torch.Tensor — Actions
action_masks : torch.Tensor | None — Boolean masks for invalid actions (True = valid).

Returns

tuple[torch.Tensor, torch.Tensor, torch.Tensor | None] — estimated value, log likelihood of taking those actions and entropy of the action distribution.

source method MaskableActorCriticPolicy.get_distribution(obs: PyTorchObs, action_masks: np.ndarray | None = None) → MaskableDistribution

Get the current policy distribution given the observations.

Parameters

obs : PyTorchObs — Observation
action_masks : np.ndarray | None — Actions' mask

Returns

MaskableDistribution — the action distribution.

source method MaskableActorCriticPolicy.predict_values(obs: PyTorchObs) → torch.Tensor

Get the estimated values according to the current policy given the observations.

Parameters

obs : PyTorchObs — Observation

Returns

torch.Tensor — the estimated values.

source class MaskableMultiInputActorCriticPolicy(observation_space: spaces.Dict, action_space: spaces.Space, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.Tanh, ortho_init: bool = True, features_extractor_class: type[BaseFeaturesExtractor] = CombinedExtractor, features_extractor_kwargs: dict[str, Any] | None = None, share_features_extractor: bool = True, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : MaskableActorCriticPolicy

MultiInputActorClass policy class for actor-critic algorithms (has both policy and value prediction).

Used by A2C, PPO and the likes.

Parameters

observation_space : spaces.Dict — Observation space (Tuple)
action_space : spaces.Space — Action space
lr_schedule : Schedule — Learning rate schedule (could be constant)
net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.
activation_fn : type[nn.Module] — Activation function
ortho_init : bool — Whether to use or not orthogonal initialization
features_extractor_class : type[BaseFeaturesExtractor] — Uses the CombinedExtractor
features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the feature extractor.
share_features_extractor : bool — If True, the features extractor is shared between the policy and value networks.
normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)
optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default
optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.
squash_output : bool — (bool) Getter for squash_output.

source class MultiInputActorCriticPolicy(observation_space: spaces.Dict, action_space: spaces.Space, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.Tanh, ortho_init: bool = True, use_sde: bool = False, log_std_init: float = 0.0, full_std: bool = True, use_expln: bool = False, squash_output: bool = False, features_extractor_class: type[BaseFeaturesExtractor] = CombinedExtractor, features_extractor_kwargs: dict[str, Any] | None = None, share_features_extractor: bool = True, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : ActorCriticPolicy

MultiInputActorClass policy class for actor-critic algorithms (has both policy and value prediction).

Used by A2C, PPO and the likes.

Parameters

observation_space : spaces.Dict — Observation space (Tuple)
action_space : spaces.Space — Action space
lr_schedule : Schedule — Learning rate schedule (could be constant)
net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.
activation_fn : type[nn.Module] — Activation function
ortho_init : bool — Whether to use or not orthogonal initialization
use_sde : bool — Whether to use State Dependent Exploration or not
log_std_init : float — Initial value for the log standard deviation
full_std : bool — Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE
use_expln : bool — Use expln() function instead of exp() to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.
squash_output : bool — Whether to squash the output using a tanh function, this allows to ensure boundaries when using gSDE.
features_extractor_class : type[BaseFeaturesExtractor] — Uses the CombinedExtractor
features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.
share_features_extractor : bool — If True, the features extractor is shared between the policy and value networks.
normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)
optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default
optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.
squash_output : bool — (bool) Getter for squash_output.