lit_sb3.distributions.sde

source module lit_sb3.distributions.sde

State Dependent Exploration (gSDE) distribution.

Classes

• StateDependentNoiseDistribution — Distribution class for using generalized State Dependent Exploration (gSDE).

source class StateDependentNoiseDistribution(action_dim: int, full_std: bool = True, use_expln: bool = False, squash_output: bool = False, learn_features: bool = False, epsilon: float = 1e-06)

Bases : Distribution

Distribution class for using generalized State Dependent Exploration (gSDE).

Paper: https://arxiv.org/abs/2005.05719.

It is used to create the noise exploration matrix and compute the log probability of an action with that noise.

Parameters

• action_dim : int — Dimension of the action space.

• full_std : bool — Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,)

• use_expln : bool — Use expln() function instead of exp() to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.

• squash_output : bool — Whether to squash the output using a tanh function, this ensures bounds are satisfied.

• learn_features : bool — Whether to learn features for gSDE or not. This will enable gradients to be backpropagated through the features latent_sde in the code.

• epsilon : float — small value to avoid NaN due to numerical imprecision.

Methods

• get_std — Get the standard deviation from the learned parameter.

• sample_weights — Sample weights for the noise exploration matrix,.

• proba_distribution_net — Create the layers and parameter that represent the distribution.

• proba_distribution — Create the distribution given its parameters (mean, std).

• log_prob

• entropy

• sample

• mode

• get_noise

• actions_from_params

• log_prob_from_params

source method StateDependentNoiseDistribution.get_std(log_std: torch.Tensor) → torch.Tensor

Get the standard deviation from the learned parameter.

(log of it by default). This ensures that the std is positive.

Parameters

• log_std : torch.Tensor — Log standard deviation of the action distribution.

source method StateDependentNoiseDistribution.sample_weights(log_std: torch.Tensor, batch_size: int = 1) → None

Sample weights for the noise exploration matrix,.

using a centered Gaussian distribution.

Parameters

• log_std : torch.Tensor — Log standard deviation of the action distribution.

• batch_size : int — Number of elements to sample.

source method StateDependentNoiseDistribution.proba_distribution_net(latent_dim: int, log_std_init: float = -2.0, latent_sde_dim: int | None = None) → tuple[nn.Module, nn.Parameter]

Create the layers and parameter that represent the distribution.

one output will be the deterministic action, the other parameter will be the standard deviation of the distribution that control the weights of the noise matrix.

Parameters

• latent_dim : int — Dimension of the last layer of the policy (before the action layer)

• log_std_init : float — Initial value for the log standard deviation

• latent_sde_dim : int | None — Dimension of the last layer of the features extractor for gSDE. By default, it is shared with the policy network.

source method StateDependentNoiseDistribution.proba_distribution(mean_actions: torch.Tensor, log_std: torch.Tensor, latent_sde: torch.Tensor) → SelfStateDependentNoiseDistribution

Create the distribution given its parameters (mean, std).

Parameters

• mean_actions : torch.Tensor — Mean of the action distribution.

• log_std : torch.Tensor — Log standard deviation of the action distribution.

• latent_sde : torch.Tensor — Latent variable for the SDE exploration noise.

source method StateDependentNoiseDistribution.log_prob(actions: torch.Tensor) → torch.Tensor

source method StateDependentNoiseDistribution.entropy() → torch.Tensor | None

source method StateDependentNoiseDistribution.sample() → torch.Tensor

source method StateDependentNoiseDistribution.mode() → torch.Tensor

source method StateDependentNoiseDistribution.get_noise(latent_sde: torch.Tensor) → torch.Tensor

source method StateDependentNoiseDistribution.actions_from_params(mean_actions: torch.Tensor, log_std: torch.Tensor, latent_sde: torch.Tensor, deterministic: bool = False) → torch.Tensor

source method StateDependentNoiseDistribution.log_prob_from_params(mean_actions: torch.Tensor, log_std: torch.Tensor, latent_sde: torch.Tensor) → tuple[torch.Tensor, torch.Tensor]