lit_sb3.algorithms.qrdqn.policies

source module lit_sb3.algorithms.qrdqn.policies

Classes

• QuantileNetwork — Quantile network for QR-DQN.

• QRDQNPolicy — Policy class with quantile and target networks for QR-DQN.

• CnnPolicy — Policy class for QR-DQN when using images as input.

• MultiInputPolicy — Policy class for QR-DQN when using dict observations as input.

source class QuantileNetwork(observation_space: spaces.Space, action_space: spaces.Discrete, features_extractor: BaseFeaturesExtractor, features_dim: int, n_quantiles: int = 200, net_arch: list[int] | None = None, activation_fn: type[nn.Module] = nn.ReLU, normalize_images: bool = True)

Bases : BasePolicy

Quantile network for QR-DQN.

Parameters

• observation_space : spaces.Space — Observation space

• action_space : spaces.Discrete — Action space

• n_quantiles : int — Number of quantiles

• net_arch : list[int] | None — The specification of the network architecture.

• activation_fn : type[nn.Module] — Activation function

• normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

Attributes

• device : torch.device — Infer which device this policy lives on by inspecting its parameters.

• squash_output : bool — (bool) Getter for squash_output.

Methods

• forward — Predict the quantiles.

source method QuantileNetwork.forward(obs: PyTorchObs) → torch.Tensor

Predict the quantiles.

Parameters

• obs : PyTorchObs — Observation

Returns

• torch.Tensor — The estimated quantiles for each action.

source class QRDQNPolicy(observation_space: spaces.Space, action_space: spaces.Discrete, lr_schedule: Schedule, n_quantiles: int = 200, net_arch: list[int] | None = None, activation_fn: type[nn.Module] = nn.ReLU, features_extractor_class: type[BaseFeaturesExtractor] = FlattenExtractor, features_extractor_kwargs: dict[str, Any] | None = None, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : BasePolicy

Policy class with quantile and target networks for QR-DQN.

Parameters

• observation_space : spaces.Space — Observation space

• action_space : spaces.Discrete — Action space

• lr_schedule : Schedule — Learning rate schedule (could be constant)

• n_quantiles : int — Number of quantiles

• net_arch : list[int] | None — The specification of the network architecture.

• activation_fn : type[nn.Module] — Activation function

• features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.

• features_extractor_kwargs : dict[str, Any] | None — Keyword arguments to pass to the features extractor.

• normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

• optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default

• optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

• device : torch.device — Infer which device this policy lives on by inspecting its parameters.

• squash_output : bool — (bool) Getter for squash_output.

Methods

• make_quantile_net

• forward

• set_training_mode — Put the policy in either training or evaluation mode.

source method QRDQNPolicy.make_quantile_net() → QuantileNetwork

source method QRDQNPolicy.forward(obs: PyTorchObs, deterministic: bool = True) → torch.Tensor

source method QRDQNPolicy.set_training_mode(mode: bool) → None

Put the policy in either training or evaluation mode.

This affects certain modules, such as batch normalisation and dropout.

Parameters

• mode : bool — if true, set to training mode, else set to evaluation mode

source class CnnPolicy(observation_space: spaces.Space, action_space: spaces.Discrete, lr_schedule: Schedule, n_quantiles: int = 200, net_arch: list[int] | None = None, activation_fn: type[nn.Module] = nn.ReLU, features_extractor_class: type[BaseFeaturesExtractor] = NatureCNN, features_extractor_kwargs: dict[str, Any] | None = None, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : QRDQNPolicy

Policy class for QR-DQN when using images as input.

Parameters

• observation_space : spaces.Space — Observation space

• action_space : spaces.Discrete — Action space

• lr_schedule : Schedule — Learning rate schedule (could be constant)

• n_quantiles : int — Number of quantiles

• net_arch : list[int] | None — The specification of the network architecture.

• activation_fn : type[nn.Module] — Activation function

• features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.

• normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

• optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default

• optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

• device : torch.device — Infer which device this policy lives on by inspecting its parameters.

• squash_output : bool — (bool) Getter for squash_output.

source class MultiInputPolicy(observation_space: spaces.Space, action_space: spaces.Discrete, lr_schedule: Schedule, n_quantiles: int = 200, net_arch: list[int] | None = None, activation_fn: type[nn.Module] = nn.ReLU, features_extractor_class: type[BaseFeaturesExtractor] = CombinedExtractor, features_extractor_kwargs: dict[str, Any] | None = None, normalize_images: bool = True, optimizer_class: type[torch.optim.Optimizer] = torch.optim.Adam, optimizer_kwargs: dict[str, Any] | None = None)

Bases : QRDQNPolicy

Policy class for QR-DQN when using dict observations as input.

Parameters

• observation_space : spaces.Space — Observation space

• action_space : spaces.Discrete — Action space

• lr_schedule : Schedule — Learning rate schedule (could be constant)

• n_quantiles : int — Number of quantiles

• net_arch : list[int] | None — The specification of the network architecture.

• activation_fn : type[nn.Module] — Activation function

• features_extractor_class : type[BaseFeaturesExtractor] — Features extractor to use.

• normalize_images : bool — Whether to normalize images or not, dividing by 255.0 (True by default)

• optimizer_class : type[torch.optim.Optimizer] — The optimizer to use, torch.optim.Adam by default

• optimizer_kwargs : dict[str, Any] | None — Additional keyword arguments, excluding the learning rate, to pass to the optimizer

Attributes

• device : torch.device — Infer which device this policy lives on by inspecting its parameters.

• squash_output : bool — (bool) Getter for squash_output.