lit_sb3.algorithms.ars

Classes

ARS — Augmented Random Search: https://arxiv.org/abs/1803.07055.
ARSLinearPolicy — Linear policy network for ARS.
ARSPolicy — Policy network for ARS.

source class ARS(policy: str | type[ARSPolicy], n_delta: int = 8, n_top: int | None = None, learning_rate: float | Schedule = 0.02, delta_std: float | Schedule = 0.05, zero_policy: bool = True, alive_bonus_offset: float = 0, n_eval_episodes: int = 1, policy_kwargs: dict[str, Any] | None = None, stats_window_size: int = 100, seed: int | None = None, verbose: int = 0, device: torch.device | str = 'cpu', _init_setup_model: bool = True)

Bases : BaseAlgorithm

Augmented Random Search: https://arxiv.org/abs/1803.07055.

Original implementation: https://github.com/modestyachts/ARS C++/Cuda Implementation: https://github.com/google-research/tiny-differentiable-simulator/ 150 LOC Numpy Implementation: https://github.com/alexis-jacq/numpy_ARS/blob/master/asr.py

Parameters

policy : str | type[ARSPolicy] — The policy to train, can be an instance of ARSPolicy, or a string from ["LinearPolicy", "MlpPolicy"]
env — The environment to train on, may be a string if registered with gym
n_delta : int — How many random perturbations of the policy to try at each update step.
n_top : int | None — How many of the top delta to use in each update step. Default is n_delta
learning_rate : float | Schedule — Float or schedule for the step size
delta_std : float | Schedule — Float or schedule for the exploration noise
zero_policy : bool — Boolean determining if the passed policy should have it's weights zeroed before training.
alive_bonus_offset : float — Constant added to the reward at each step, used to cancel out alive bonuses.
n_eval_episodes : int — Number of episodes to evaluate each candidate.
policy_kwargs : dict[str, Any] | None — Keyword arguments to pass to the policy on creation. See :ref:ars_policies
stats_window_size : int — Window size for the rollout logging, specifying the number of episodes to average the reported success rate, mean episode length, and mean reward over
seed : int | None — Random seed for the training
verbose : int — Verbosity level: 0 no output, 1 info, 2 debug
device : torch.device | str — Torch device to use for training, defaults to "cpu"
_init_setup_model : bool — Whether or not to build the network at the creation of the instance

Attributes

device : torch.device — Return current device (Lightning-aware).
hparams : Union[AttributeDict, MutableMapping] — meth:save_hyperparameters. It is mutable by the user. For the frozen set of initial hyperparameters, use :attr:hparams_initial.
hparams_initial : AttributeDict — meth:save_hyperparameters. These contents are read-only. Manual updates to the saved hyperparameters can instead be performed through :attr:hparams.
example_input_array : Optional[Union[Tensor, tuple, dict]] — meth:forward method. The return type is interpreted as follows:
current_epoch : int — The current epoch in the Trainer, or 0 if not attached.
global_step : int — Total training batches seen across all epochs.
global_rank : int — The index of the current process across all nodes and devices.
local_rank : int — The index of the current process within a single node.
on_gpu : bool — Returns True if this model is currently located on a GPU.
automatic_optimization : bool — If set to False you are responsible for calling .backward(), .step(), .zero_grad().
strict_loading : bool — Determines how Lightning loads this model using .load_state_dict(..., strict=model.strict_loading).
logger : Optional[Union[Logger, FabricLogger]] — Reference to the logger object in the Trainer.
loggers : Union[list[Logger], list[FabricLogger]] — Reference to the list of loggers in the Trainer.
device_mesh : Optional['DeviceMesh'] — Strategies like ModelParallelStrategy will create a device mesh that can be accessed in the :meth:~lightning.pytorch.core.hooks.ModelHooks.configure_model hook to parallelize the LightningModule.
env : VecEnv | None — Return the active training env from the trainer, or None.

Methods

evaluate_candidates — Evaluate each candidate.
dump_logs — Dump information to the logger.
learn — Return a trained model.
set_parameters

source method ARS.evaluate_candidates(candidate_weights: torch.Tensor, callback: BaseCallback, async_eval: AsyncEval | None) → torch.Tensor

Evaluate each candidate.

Parameters

candidate_weights : torch.Tensor — The candidate weights to be evaluated.
callback : BaseCallback — Callback that will be called at each step (or after evaluation in the multiprocess version)
async_eval : AsyncEval | None — The object for asynchronous evaluation of candidates.

Returns

torch.Tensor — The episodic return for each candidate.

source method ARS.dump_logs() → None

Dump information to the logger.

source method ARS.learn(total_timesteps: int, env: GymEnv | str | None = None, callback: MaybeCallback = None, log_every_n_steps: int | None = 1, tb_log_name: str = 'ARS', reset_num_timesteps: bool = True, async_eval: AsyncEval | None = None, progress_bar: bool = False, default_root_dir: str | Path | None = None) → SelfARS

Return a trained model.

Parameters

total_timesteps : int — The total number of samples (env steps) to train on
env : GymEnv | str | None — Training environment (gym env, env-id string, or VecEnv).
callback : MaybeCallback — callback(s) called at every step with state of the algorithm.
log_every_n_steps : int | None — The number of timesteps before logging.
tb_log_name : str — the name of the run for TensorBoard logging
reset_num_timesteps : bool — whether or not to reset the current timestep number (used in logging)
async_eval : AsyncEval | None — The object for asynchronous evaluation of candidates.
progress_bar : bool — Display a progress bar using tqdm and rich.
default_root_dir : str | Path | None — Unused for ARS; accepted for API compatibility.

Returns

SelfARS — the trained model

Raises

TypeError

source method ARS.set_parameters(load_path_or_dict: str | dict[str, dict], exact_match: bool = True, device: torch.device | str = 'auto') → None

source class ARSLinearPolicy(observation_space: spaces.Space, action_space: spaces.Space, with_bias: bool = False, squash_output: bool = False)

Bases : ARSPolicy

Linear policy network for ARS.

Parameters

observation_space : spaces.Space — The observation space of the environment
action_space : spaces.Space — The action space of the environment
with_bias : bool — With or without bias on the output
squash_output : bool — For continuous actions, whether the output is squashed or not using a tanh() function. If not squashed with tanh the output will instead be clipped.

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.
squash_output : bool — (bool) Getter for squash_output.

source class ARSPolicy(observation_space: spaces.Space, action_space: spaces.Space, net_arch: list[int] | None = None, activation_fn: type[nn.Module] = nn.ReLU, with_bias: bool = True, squash_output: bool = True)

Bases : BasePolicy

Policy network for ARS.

Parameters

observation_space : spaces.Space — The observation space of the environment
action_space : spaces.Space — The action space of the environment
net_arch : list[int] | None — Network architecture, defaults to a 2 layers MLP with 64 hidden nodes.
activation_fn : type[nn.Module] — Activation function
with_bias : bool — If set to False, the layers will not learn an additive bias
squash_output : bool — For continuous actions, whether the output is squashed or not using a tanh() function. If not squashed with tanh the output will instead be clipped.

Attributes

device : torch.device — Infer which device this policy lives on by inspecting its parameters.
squash_output : bool — (bool) Getter for squash_output.

Methods

forward

source method ARSPolicy.forward(obs: PyTorchObs) → torch.Tensor

Raises

NotImplementedError