lit_sb3.algorithms.crossq.policies

source module lit_sb3.algorithms.crossq.policies

Classes

• Actor — Actor network (policy) for CrossQ.

• CrossQCritic — Critic network(s) for CrossQ.

• CrossQPolicy — Policy class (with both actor and critic) for CrossQ.

source class Actor(observation_space: spaces.Space, action_space: spaces.Box, net_arch: list[int], features_extractor: nn.Module, features_dim: int, activation_fn: type[nn.Module] = nn.ReLU, use_sde: bool = False, log_std_init: float = -3, full_std: bool = True, use_expln: bool = False, clip_mean: float = 2.0, normalize_images: bool = True, batch_norm: bool = True, batch_norm_momentum: float = 0.01, batch_norm_eps: float = 0.001, renorm_warmup_steps: int = 100000)

Bases : BasePolicy

Actor network (policy) for CrossQ.

It contains BatchRenorm layers to stabilize and accelerate training.

Parameters

• observation_space : spaces.Space — Obervation space

• action_space : spaces.Box — Action space

• net_arch : list[int] — Network architecture

• features_extractor : nn.Module — Network to extract features (a CNN when using images, a nn.Flatten() layer otherwise)

• features_dim : int — Number of features

• activation_fn : type[nn.Module] — Activation function

• use_sde : bool — Whether to use State Dependent Exploration or not

• log_std_init : float — Initial value for the log standard deviation

• full_std : bool — Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE.

• use_expln : bool — Use expln() function instead of exp() when using gSDE to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.

• clip_mean : float — Clip the mean output when using gSDE to avoid numerical instability.

• normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

• batch_norm : bool — Whether to use Batch Renorm layers (default=True)

• batch_norm_momentum : float — The rate of convergence for the batch renormalization statistics

• batch_norm_eps : float — A small value added to the variance to prevent division by zero

• renorm_warmup_steps : int — Number of steps to warm up BatchRenorm statistics before the running statistics are used for normalization.

Attributes

• device : torch.device — Infer which device this policy lives on by inspecting its parameters.

• squash_output : bool — (bool) Getter for squash_output.

Methods

• get_std — Retrieve the standard deviation of the action distribution.

• reset_noise — Sample new weights for the exploration matrix, when using gSDE.

• get_action_dist_params — Get the parameters for the action distribution.

• forward

• action_log_prob

• set_bn_training_mode — Set the training mode of the BatchRenorm layers.

source method Actor.get_std() → torch.Tensor

Retrieve the standard deviation of the action distribution.

Only useful when using gSDE. It corresponds to torch.exp(log_std) in the normal case, but is slightly different when using expln function (cf StateDependentNoiseDistribution doc).

Returns

• torch.Tensor — Standard deviation of the action dist when available.

source method Actor.reset_noise(batch_size: int = 1) → None

Sample new weights for the exploration matrix, when using gSDE.

Parameters

• batch_size : int

• batch_size : int — The number of samples in the batch.

source method Actor.get_action_dist_params(obs: PyTorchObs) → tuple[torch.Tensor, torch.Tensor, dict[str, torch.Tensor]]

Get the parameters for the action distribution.

Parameters

• obs : PyTorchObs

• obs : PyTorchObs — The current observation.

Returns

• tuple[torch.Tensor, torch.Tensor, dict[str, torch.Tensor]] — Mean, standard deviation and optional keyword arguments.

source method Actor.forward(obs: PyTorchObs, deterministic: bool = False) → torch.Tensor

source method Actor.action_log_prob(obs: PyTorchObs) → tuple[torch.Tensor, torch.Tensor]

source method Actor.set_bn_training_mode(mode: bool) → None

Set the training mode of the BatchRenorm layers.

When training is True, the running statistics are updated.

Parameters

• mode : bool — Whether to set the layers in training mode or not

source class CrossQCritic(observation_space: spaces.Space, action_space: spaces.Box, net_arch: list[int], features_extractor: BaseFeaturesExtractor, features_dim: int, activation_fn: type[nn.Module] = nn.ReLU, normalize_images: bool = True, n_critics: int = 2, share_features_extractor: bool = True, batch_norm: bool = True, batch_norm_momentum: float = 0.01, batch_norm_eps: float = 0.001, renorm_warmup_steps: int = 100000)

Bases : BaseModel

Critic network(s) for CrossQ.

The difference with standard critic networks used by SAC/TD3 is that it uses BatchRenorm layers.

By default, it creates two critic networks used to reduce overestimation thanks to clipped Q-learning (cf TD3 paper).

Parameters

• observation_space : spaces.Space — Obervation space

• action_space : spaces.Box — Action space

• net_arch : list[int] — Network architecture

• features_extractor : BaseFeaturesExtractor — Network to extract features (a CNN when using images, a nn.Flatten() layer otherwise)

• features_dim : int — Number of features

• activation_fn : type[nn.Module] — Activation function

• normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

• n_critics : int — Number of critic networks to create.

• share_features_extractor : bool — Whether the features extractor is shared or not between the actor and the critic (this saves computation time)

• batch_norm : bool — Whether to use Batch Renorm layers (default=True)

• batch_norm_momentum : float — The rate of convergence for the batch renormalization statistics

• batch_norm_eps : float — A small value added to the variance to prevent division by zero

• renorm_warmup_steps : int — Number of steps to warm up BatchRenorm statistics before the running statistics are used for normalization.

Attributes

• device : torch.device — Infer which device this policy lives on by inspecting its parameters.

Methods

• forward

• set_bn_training_mode — Set the training mode of the BatchRenorm layers.

source method CrossQCritic.forward(obs: torch.Tensor, actions: torch.Tensor) → tuple[torch.Tensor, ...]

source method CrossQCritic.set_bn_training_mode(mode: bool) → None

Set the training mode of the BatchRenorm layers.

When training is True, the running statistics are updated.

Parameters

• mode : bool — Whether to set the layers in training mode or not

source class CrossQPolicy(observation_space: spaces.Space, action_space: spaces.Box, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.ReLU, batch_norm: bool = True, batch_norm_momentum: float = 0.01, batch_norm_eps: float = 0.001, renorm_warmup_steps: int = 100000, use_sde: bool = False, log_std_init: float = -3, use_expln: bool = False, clip_mean: float = 2.0, features_extractor_class: type[BaseFeaturesExtractor] = FlattenExtractor, features_extractor_kwargs: dict[str, Any] | None = None, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None, n_critics: int = 2, share_features_extractor: bool = False)

Bases : BasePolicy

Policy class (with both actor and critic) for CrossQ.

Parameters

• observation_space : spaces.Space — Observation space

• action_space : spaces.Box — Action space

• lr_schedule : Schedule — Learning rate schedule (could be constant)

• net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.

• activation_fn : type[nn.Module] — Activation function

• batch_norm : bool — Whether to use Batch Renorm layers (default=True)

• batch_norm_momentum : float — The rate of convergence for the batch renormalization statistics

• batch_norm_eps : float — A small value added to the variance to prevent division by zero

• renorm_warmup_steps : int — Number of steps to warm up BatchRenorm statistics before the running statistics are used for normalization.

• use_sde : bool — Whether to use State Dependent Exploration or not

• log_std_init : float — Initial value for the log standard deviation

• use_expln : bool — Use expln() function instead of exp() when using gSDE to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.

• clip_mean : float — Clip the mean output when using gSDE to avoid numerical instability.

• features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.

• features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.

• normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

• optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default

• optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

• n_critics : int — Number of critic networks to create.

• share_features_extractor : bool — Whether to share or not the features extractor between the actor and the critic (this saves computation time)

Attributes

• device : torch.device — Infer which device this policy lives on by inspecting its parameters.

• squash_output : bool — (bool) Getter for squash_output.

Methods

• reset_noise — Sample new weights for the exploration matrix, when using gSDE.

• make_actor

• make_critic

• forward

• set_training_mode — Put the policy in either training or evaluation mode.

source method CrossQPolicy.reset_noise(batch_size: int = 1) → None

Sample new weights for the exploration matrix, when using gSDE.

Parameters

• batch_size : int

• batch_size : int — The number of samples in the batch.

source method CrossQPolicy.make_actor(features_extractor: BaseFeaturesExtractor | None = None) → Actor

source method CrossQPolicy.make_critic(features_extractor: BaseFeaturesExtractor | None = None) → CrossQCritic

source method CrossQPolicy.forward(obs: PyTorchObs, deterministic: bool = False) → torch.Tensor

source method CrossQPolicy.set_training_mode(mode: bool) → None

Put the policy in either training or evaluation mode.

This affects certain modules, such as batch normalisation and dropout.

Parameters

• mode : bool — if true, set to training mode, else set to evaluation mode