Skip to content

lit_sb3.algorithms.ddpg

source package lit_sb3.algorithms.ddpg

Classes

  • CnnPolicy — Policy class (with both actor and critic) for TD3.

  • DDPG — Deep Deterministic Policy Gradient (DDPG).

  • MultiInputPolicy — Policy class (with both actor and critic) for TD3 to be used with Dict observation spaces.

source class CnnPolicy(observation_space: spaces.Space, action_space: spaces.Box, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.ReLU, features_extractor_class: type[BaseFeaturesExtractor] = NatureCNN, features_extractor_kwargs: dict[str, Any] | None = None, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None, n_critics: int = 2, share_features_extractor: bool = False)

Bases : TD3Policy

Policy class (with both actor and critic) for TD3.

Parameters

  • observation_space : spaces.Space — Observation space

  • action_space : spaces.Box — Action space

  • lr_schedule : Schedule — Learning rate schedule (could be constant)

  • net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.

  • activation_fn : type[nn.Module] — Activation function

  • features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.

  • features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.

  • normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

  • optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default

  • optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

  • n_critics : int — Number of critic networks to create.

  • share_features_extractor : bool — Whether to share or not the features extractor between the actor and the critic (this saves computation time)

Attributes

  • device : torch.device — Infer which device this policy lives on by inspecting its parameters.

  • squash_output : bool — (bool) Getter for squash_output.

source class DDPG(policy: str | type[TD3Policy], learning_rate: float | Schedule = 0.001, buffer_size: int = 1000000, learning_starts: int = 100, batch_size: int = 256, tau: float = 0.005, gamma: float = 0.99, train_freq: int | tuple[int, str] = 1, gradient_steps: int = 1, action_noise: ActionNoise | None = None, replay_buffer_class: type[ReplayBuffer] | None = None, replay_buffer_kwargs: dict[str, Any] | None = None, optimize_memory_usage: bool = False, n_steps: int = 1, policy_kwargs: dict[str, Any] | None = None, verbose: int = 0, seed: int | None = None, device: torch.device | str = 'auto', _init_setup_model: bool = True)

Bases : TD3

Deep Deterministic Policy Gradient (DDPG).

Deterministic Policy Gradient: http://proceedings.mlr.press/v32/silver14.pdf DDPG Paper: https://arxiv.org/abs/1509.02971 Introduction to DDPG: https://spinningup.openai.com/en/latest/algorithms/ddpg.html

Note: we treat DDPG as a special case of its successor TD3.

Parameters

  • policy : str | type[TD3Policy] — The policy model to use (MlpPolicy, CnnPolicy, ...)

  • env — The environment to learn from (if registered in Gym, can be str)

  • learning_rate : float | Schedule — learning rate for adam optimizer, the same learning rate will be used for all networks (Q-Values, Actor and Value function) it can be a function of the current progress remaining (from 1 to 0)

  • buffer_size : int — size of the replay buffer

  • learning_starts : int — how many steps of the model to collect transitions for before learning starts

  • batch_size : int — Minibatch size for each gradient update

  • tau : float — the soft update coefficient ("Polyak update", between 0 and 1)

  • gamma : float — the discount factor

  • train_freq : int | tuple[int, str] — Update the model every train_freq steps. Alternatively pass a tuple of frequency and unit like (5, "step") or (2, "episode").

  • gradient_steps : int — How many gradient steps to do after each rollout (see train_freq) Set to -1 means to do as many gradient steps as steps done in the environment during the rollout.

  • action_noise : ActionNoise | None — the action noise type (None by default), this can help for hard exploration problem. Cf common.noise for the different action noise type.

  • replay_buffer_class : type[ReplayBuffer] | None — Replay buffer class to use (for instance HerReplayBuffer). If None, it will be automatically selected.

  • replay_buffer_kwargs : dict[str, Any] | None — Keyword arguments to pass to the replay buffer on creation.

  • optimize_memory_usage : bool — Enable a memory efficient variant of the replay buffer at a cost of more complexity. See https://github.com/DLR-RM/stable-baselines3/issues/37#issuecomment-637501195

  • n_steps : int — When n_step > 1, uses n-step return (with the NStepReplayBuffer) when updating the Q-value network.

  • policy_kwargs : dict[str, Any] | None — additional arguments to be passed to the policy on creation. See :ref:ddpg_policies

  • verbose : int — Verbosity level: 0 for no output, 1 for info messages (such as device or wrappers used), 2 for debug messages

  • seed : int | None — Seed for the pseudo random generators

  • device : torch.device | str — Device (cpu, cuda, ...) on which the code should be run. Setting it to auto, the code will be run on the GPU if possible.

  • _init_setup_model : bool — Whether or not to build the network at the creation of the instance

Attributes

  • device : torch.device — Return current device (Lightning-aware).

  • hparams : Union[AttributeDict, MutableMapping] — meth:save_hyperparameters. It is mutable by the user. For the frozen set of initial hyperparameters, use :attr:hparams_initial.

  • hparams_initial : AttributeDict — meth:save_hyperparameters. These contents are read-only. Manual updates to the saved hyperparameters can instead be performed through :attr:hparams.

  • example_input_array : Optional[Union[Tensor, tuple, dict]] — meth:forward method. The return type is interpreted as follows:

  • current_epoch : int — The current epoch in the Trainer, or 0 if not attached.

  • global_step : int — Total training batches seen across all epochs.

  • global_rank : int — The index of the current process across all nodes and devices.

  • local_rank : int — The index of the current process within a single node.

  • on_gpu : bool — Returns True if this model is currently located on a GPU.

  • strict_loading : bool — Determines how Lightning loads this model using .load_state_dict(..., strict=model.strict_loading).

  • logger : Optional[Union[Logger, FabricLogger]] — Reference to the logger object in the Trainer.

  • loggers : Union[list[Logger], list[FabricLogger]] — Reference to the list of loggers in the Trainer.

  • device_mesh : Optional['DeviceMesh'] — Strategies like ModelParallelStrategy will create a device mesh that can be accessed in the :meth:~lightning.pytorch.core.hooks.ModelHooks.configure_model hook to parallelize the LightningModule.

  • env : VecEnv | None — Return the active training env from the trainer, or None.

Methods

source method DDPG.learn(total_timesteps: int, env: GymEnv | str | None = None, callback: MaybeCallback = None, log_every_n_steps: int | None = 4, tb_log_name: str = 'DDPG', reset_num_timesteps: bool = True, progress_bar: bool = False) → SelfDDPG

source class MultiInputPolicy(observation_space: spaces.Dict, action_space: spaces.Box, lr_schedule: Schedule, net_arch: list[int] | dict[str, list[int]] | None = None, activation_fn: type[nn.Module] = nn.ReLU, features_extractor_class: type[BaseFeaturesExtractor] = CombinedExtractor, features_extractor_kwargs: dict[str, Any] | None = None, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None, n_critics: int = 2, share_features_extractor: bool = False)

Bases : TD3Policy

Policy class (with both actor and critic) for TD3 to be used with Dict observation spaces.

Parameters

  • observation_space : spaces.Dict — Observation space

  • action_space : spaces.Box — Action space

  • lr_schedule : Schedule — Learning rate schedule (could be constant)

  • net_arch : list[int] | dict[str, list[int]] | None — The specification of the policy and value networks.

  • activation_fn : type[nn.Module] — Activation function

  • features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.

  • features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.

  • normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

  • optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default

  • optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

  • n_critics : int — Number of critic networks to create.

  • share_features_extractor : bool — Whether to share or not the features extractor between the actor and the critic (this saves computation time)

Attributes

  • device : torch.device — Infer which device this policy lives on by inspecting its parameters.

  • squash_output : bool — (bool) Getter for squash_output.