lit_sb3.algorithms.ppo

source package lit_sb3.algorithms.ppo

Classes

• PPO — Proximal Policy Optimization algorithm (PPO) (clip version).

source class PPO(policy: str | type[ActorCriticPolicy], learning_rate: float | Schedule = 0.0003, n_steps: int = 2048, batch_size: int = 64, n_epochs: int = 10, gamma: float = 0.99, gae_lambda: float = 0.95, clip_range: float | Schedule = 0.2, clip_range_vf: None | float | Schedule = None, normalize_advantage: bool = True, ent_coef: float = 0.0, vf_coef: float = 0.5, max_grad_norm: float = 0.5, use_sde: bool = False, sde_sample_freq: int = -1, rollout_buffer_class: type[RolloutBuffer] | None = None, rollout_buffer_kwargs: dict[str, Any] | None = None, target_kl: float | None = None, stats_window_size: int = 100, policy_kwargs: dict[str, Any] | None = None, verbose: int = 0, seed: int | None = None, device: torch.device | str = 'auto', _init_setup_model: bool = True)

Bases : OnPolicyAlgorithm

Proximal Policy Optimization algorithm (PPO) (clip version).

Paper: https://arxiv.org/abs/1707.06347 Code: This implementation borrows code from OpenAI Spinning Up (https://github.com/openai/spinningup/) https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail and Stable Baselines (PPO2 from https://github.com/hill-a/stable-baselines)

Introduction to PPO: https://spinningup.openai.com/en/latest/algorithms/ppo.html

Parameters

• policy : str | type[ActorCriticPolicy] — The policy model to use (MlpPolicy, CnnPolicy, ...)

• env — The environment to learn from (if registered in Gym, can be str)

• learning_rate : float | Schedule — The learning rate, it can be a function of the current progress remaining (from 1 to 0)

• n_steps : int — The number of steps to run for each environment per update (i.e. rollout buffer size is n_steps * n_envs where n_envs is number of environment copies running in parallel) NOTE: n_steps * n_envs must be greater than 1 (because of the advantage normalization) See https://github.com/pytorch/pytorch/issues/29372

• batch_size : int — Minibatch size

• n_epochs : int — Number of epoch when optimizing the surrogate loss

• gamma : float — Discount factor

• gae_lambda : float — Factor for trade-off of bias vs variance for Generalized Advantage Estimator

• clip_range : float | Schedule — Clipping parameter, it can be a function of the current progress remaining (from 1 to 0).

• clip_range_vf : None | float | Schedule — Clipping parameter for the value function, it can be a function of the current progress remaining (from 1 to 0). This is a parameter specific to the OpenAI implementation. If None is passed (default), no clipping will be done on the value function. IMPORTANT: this clipping depends on the reward scaling.

• normalize_advantage : bool — Whether to normalize or not the advantage

• ent_coef : float — Entropy coefficient for the loss calculation

• vf_coef : float — Value function coefficient for the loss calculation

• max_grad_norm : float — The maximum value for the gradient clipping

• use_sde : bool — Whether to use generalized State Dependent Exploration (gSDE) instead of action noise exploration (default: False)

• sde_sample_freq : int — Sample a new noise matrix every n steps when using gSDE Default: -1 (only sample at the beginning of the rollout)

• rollout_buffer_class : type[RolloutBuffer] | None — Rollout buffer class to use. If None, it will be automatically selected.

• rollout_buffer_kwargs : dict[str, Any] | None — Keyword arguments to pass to the rollout buffer on creation

• target_kl : float | None — Limit the KL divergence between updates, because the clipping is not enough to prevent large update see issue #213 (cf https://github.com/hill-a/stable-baselines/issues/213) By default, there is no limit on the kl div.

• stats_window_size : int — Window size for the rollout logging, specifying the number of episodes to average the reported success rate, mean episode length, and mean reward over

• tensorboard_log — the log location for tensorboard (if None, no logging)

• policy_kwargs : dict[str, Any] | None — additional arguments to be passed to the policy on creation. See :ref:ppo_policies

• verbose : int — Verbosity level: 0 for no output, 1 for info messages (such as device or wrappers used), 2 for debug messages

• seed : int | None — Seed for the pseudo random generators

• device : torch.device | str — Device (cpu, cuda, ...) on which the code should be run. Setting it to auto, the code will be run on the GPU if possible.

• _init_setup_model : bool — Whether or not to build the network at the creation of the instance

Attributes

• device : torch.device — Return current device (Lightning-aware).

• hparams : Union[AttributeDict, MutableMapping] — meth:save_hyperparameters. It is mutable by the user. For the frozen set of initial hyperparameters, use :attr:hparams_initial.

• hparams_initial : AttributeDict — meth:save_hyperparameters. These contents are read-only. Manual updates to the saved hyperparameters can instead be performed through :attr:hparams.

• example_input_array : Optional[Union[Tensor, tuple, dict]] — meth:forward method. The return type is interpreted as follows:

• current_epoch : int — The current epoch in the Trainer, or 0 if not attached.

• global_step : int — Total training batches seen across all epochs.

• global_rank : int — The index of the current process across all nodes and devices.

• local_rank : int — The index of the current process within a single node.

• on_gpu : bool — Returns True if this model is currently located on a GPU.

• automatic_optimization : bool — If set to False you are responsible for calling .backward(), .step(), .zero_grad().

• strict_loading : bool — Determines how Lightning loads this model using .load_state_dict(..., strict=model.strict_loading).

• logger : Optional[Union[Logger, FabricLogger]] — Reference to the logger object in the Trainer.

• loggers : Union[list[Logger], list[FabricLogger]] — Reference to the list of loggers in the Trainer.

• device_mesh : Optional['DeviceMesh'] — Strategies like ModelParallelStrategy will create a device mesh that can be accessed in the :meth:~lightning.pytorch.core.hooks.ModelHooks.configure_model hook to parallelize the LightningModule.

• env : VecEnv | None — Return the active training env from the trainer, or None.

Methods

• configure_optimizers — Return the policy optimizer.

• on_train_epoch_start — Run base rollout collection hook then set policy to train mode.

• training_step — Compute PPO clipped-surrogate loss for one mini-batch of rollout data.

• on_train_epoch_end — Log epoch-level PPO stats.

• learn

source method PPO.configure_optimizers() → OptimizerLRScheduler

Return the policy optimizer.

source method PPO.on_train_epoch_start() → None

Run base rollout collection hook then set policy to train mode.

source method PPO.training_step(batch: Any, batch_idx: int) → torch.Tensor

Compute PPO clipped-surrogate loss for one mini-batch of rollout data.

source method PPO.on_train_epoch_end() → None

Log epoch-level PPO stats.

source method PPO.learn(total_timesteps: int, env: GymEnv | str | None = None, callback: MaybeCallback = None, log_every_n_steps: int | None = 1, tb_log_name: str = 'PPO', reset_num_timesteps: bool = True, progress_bar: bool = False, default_root_dir: str | Path | None = None) → SelfPPO