Skip to content

lit_sb3.algorithms.a2c

source package lit_sb3.algorithms.a2c

Classes

  • A2C — Advantage Actor Critic (A2C).

source class A2C(policy: str | type[ActorCriticPolicy], learning_rate: float | Schedule = 0.0007, n_steps: int = 5, gamma: float = 0.99, gae_lambda: float = 1.0, ent_coef: float = 0.0, vf_coef: float = 0.5, max_grad_norm: float = 0.5, rms_prop_eps: float = 1e-05, use_rms_prop: bool = True, use_sde: bool = False, sde_sample_freq: int = -1, rollout_buffer_class: type[RolloutBuffer] | None = None, rollout_buffer_kwargs: dict[str, Any] | None = None, normalize_advantage: bool = False, stats_window_size: int = 100, policy_kwargs: dict[str, Any] | None = None, verbose: int = 0, seed: int | None = None, device: torch.device | str = 'auto', _init_setup_model: bool = True)

Bases : OnPolicyAlgorithm

Advantage Actor Critic (A2C).

Paper: https://arxiv.org/abs/1602.01783 Code: This implementation borrows code from https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail and and Stable Baselines (https://github.com/hill-a/stable-baselines)

Introduction to A2C: https://hackernoon.com/intuitive-rl-intro-to-advantage-actor-critic-a2c-4ff545978752

Parameters

  • policy : str | type[ActorCriticPolicy] — The policy model to use (MlpPolicy, CnnPolicy, ...)

  • env — The environment to learn from (if registered in Gym, can be str)

  • learning_rate : float | Schedule — The learning rate, it can be a function of the current progress remaining (from 1 to 0)

  • n_steps : int — The number of steps to run for each environment per update (i.e. batch size is n_steps * n_env where n_env is number of environment copies running in parallel)

  • gamma : float — Discount factor

  • gae_lambda : float — Factor for trade-off of bias vs variance for Generalized Advantage Estimator. Equivalent to classic advantage when set to 1.

  • ent_coef : float — Entropy coefficient for the loss calculation

  • vf_coef : float — Value function coefficient for the loss calculation

  • max_grad_norm : float — The maximum value for the gradient clipping

  • rms_prop_eps : float — RMSProp epsilon. It stabilizes square root computation in denominator of RMSProp update

  • use_rms_prop : bool — Whether to use RMSprop (default) or Adam as optimizer

  • use_sde : bool — Whether to use generalized State Dependent Exploration (gSDE) instead of action noise exploration (default: False)

  • sde_sample_freq : int — Sample a new noise matrix every n steps when using gSDE Default: -1 (only sample at the beginning of the rollout)

  • rollout_buffer_class : type[RolloutBuffer] | None — Rollout buffer class to use. If None, it will be automatically selected.

  • rollout_buffer_kwargs : dict[str, Any] | None — Keyword arguments to pass to the rollout buffer on creation.

  • normalize_advantage : bool — Whether to normalize or not the advantage

  • stats_window_size : int — Window size for the rollout logging, specifying the number of episodes to average the reported success rate, mean episode length, and mean reward over

  • tensorboard_log — the log location for tensorboard (if None, no logging)

  • policy_kwargs : dict[str, Any] | None — additional arguments to be passed to the policy on creation. See :ref:a2c_policies

  • verbose : int — Verbosity level: 0 for no output, 1 for info messages (such as device or wrappers used), 2 for debug messages

  • seed : int | None — Seed for the pseudo random generators

  • device : torch.device | str — Device (cpu, cuda, ...) on which the code should be run. Setting it to auto, the code will be run on the GPU if possible.

  • _init_setup_model : bool — Whether or not to build the network at the creation of the instance

Attributes

  • device : torch.device — Return current device (Lightning-aware).

  • hparams : Union[AttributeDict, MutableMapping] — meth:save_hyperparameters. It is mutable by the user. For the frozen set of initial hyperparameters, use :attr:hparams_initial.

  • hparams_initial : AttributeDict — meth:save_hyperparameters. These contents are read-only. Manual updates to the saved hyperparameters can instead be performed through :attr:hparams.

  • example_input_array : Optional[Union[Tensor, tuple, dict]] — meth:forward method. The return type is interpreted as follows:

  • current_epoch : int — The current epoch in the Trainer, or 0 if not attached.

  • global_step : int — Total training batches seen across all epochs.

  • global_rank : int — The index of the current process across all nodes and devices.

  • local_rank : int — The index of the current process within a single node.

  • on_gpu : bool — Returns True if this model is currently located on a GPU.

  • automatic_optimization : bool — If set to False you are responsible for calling .backward(), .step(), .zero_grad().

  • strict_loading : bool — Determines how Lightning loads this model using .load_state_dict(..., strict=model.strict_loading).

  • logger : Optional[Union[Logger, FabricLogger]] — Reference to the logger object in the Trainer.

  • loggers : Union[list[Logger], list[FabricLogger]] — Reference to the list of loggers in the Trainer.

  • device_mesh : Optional['DeviceMesh'] — Strategies like ModelParallelStrategy will create a device mesh that can be accessed in the :meth:~lightning.pytorch.core.hooks.ModelHooks.configure_model hook to parallelize the LightningModule.

  • env : VecEnv | None — Return the active training env from the trainer, or None.

Methods

source method A2C.configure_optimizers() → OptimizerLRScheduler

source method A2C.on_train_epoch_start() → None

source method A2C.training_step(batch: Any, batch_idx: int) → torch.Tensor

A2C loss on the full rollout buffer (single gradient step).

source method A2C.on_train_epoch_end() → None

source method A2C.learn(total_timesteps: int, env: GymEnv | str | None = None, callback: MaybeCallback = None, log_every_n_steps: int | None = 100, tb_log_name: str = 'A2C', reset_num_timesteps: bool = True, progress_bar: bool = False, default_root_dir: str | Path | None = None) → SelfA2C