Skip to content

lit_sb3.policies.continuous_critic

source module lit_sb3.policies.continuous_critic

Policies: abstract base class and concrete implementations.

Classes

source class ContinuousCritic(observation_space: spaces.Space, action_space: spaces.Box, net_arch: list[int], features_extractor: BaseFeaturesExtractor, features_dim: int, activation_fn: type[nn.Module] = nn.ReLU, normalize_images: bool = True, n_critics: int = 2, share_features_extractor: bool = True)

Bases : BaseModel

Critic network(s) for DDPG/SAC/TD3.

It represents the action-state value function (Q-value function). Compared to A2C/PPO critics, this one represents the Q-value and takes the continuous action as input. It is concatenated with the state and then fed to the network which outputs a single value: Q(s, a). For more recent algorithms like SAC/TD3, multiple networks are created to give different estimates.

By default, it creates two critic networks used to reduce overestimation thanks to clipped Q-learning (cf TD3 paper).

Parameters

  • observation_space : spaces.Space — Observation space

  • action_space : spaces.Box — Action space

  • net_arch : list[int] — Network architecture

  • features_extractor : BaseFeaturesExtractor — Network to extract features (a CNN when using images, a nn.Flatten() layer otherwise)

  • features_dim : int — Number of features

  • activation_fn : type[nn.Module] — Activation function

  • normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

  • n_critics : int — Number of critic networks to create.

  • share_features_extractor : bool — Whether the features extractor is shared or not between the actor and the critic (this saves computation time)

Attributes

  • device : torch.device — Infer which device this policy lives on by inspecting its parameters.

Methods

source method ContinuousCritic.forward(obs: torch.Tensor, actions: torch.Tensor) → tuple[torch.Tensor, ...]

source method ContinuousCritic.q1_forward(obs: torch.Tensor, actions: torch.Tensor) → torch.Tensor

Only predict the Q-value using the first network.

This allows to reduce computation when all the estimates are not needed (e.g. when updating the policy in TD3).