TD3 on LunarLanderContinuous-v2¶

Twin Delayed DDPG (TD3) addresses overestimation bias in actor-critic methods through three key improvements over DDPG:

1. Twin critics: two Q-networks; the minimum is used to compute the target, reducing overestimation.
2. Delayed policy updates: the actor and target networks update less frequently than the critics (policy_delay).
3. Target policy smoothing: noise is added to the target action to smooth the value function estimate.

Paper: Addressing Function Approximation Error in Actor-Critic Methods

In this notebook we train TD3 on LunarLanderContinuous-v2 using lit-sb3.

Setup¶

In [ ]:

import warnings

warnings.filterwarnings("ignore")

import gymnasium as gym
import numpy as np

from lit_sb3 import RLEnvDataModule, RLTrainer
from lit_sb3.algorithms.td3.td3 import TD3
from lit_sb3.core.noise import NormalActionNoise

import warnings warnings.filterwarnings("ignore") import gymnasium as gym import numpy as np from lit_sb3 import RLEnvDataModule, RLTrainer from lit_sb3.algorithms.td3.td3 import TD3 from lit_sb3.core.noise import NormalActionNoise

Action noise¶

Similar to DDPG, TD3 uses a deterministic policy and requires exploration noise during data collection.

In [ ]:

env = gym.make("LunarLanderContinuous-v2")
n_actions = env.action_space.shape[0]  # ty:ignore[not-subscriptable]
action_noise = NormalActionNoise(
    mean=np.zeros(n_actions),
    sigma=0.1 * np.ones(n_actions),
)
env.close()

env = gym.make("LunarLanderContinuous-v2") n_actions = env.action_space.shape[0] # ty:ignore[not-subscriptable] action_noise = NormalActionNoise( mean=np.zeros(n_actions), sigma=0.1 * np.ones(n_actions), ) env.close()

Create the model¶

Key hyperparameters:

Parameter	Description
policy_delay	Actor / target-network update frequency (every N critic steps)
target_policy_noise	Std of Gaussian smoothing noise added to target actions
target_noise_clip	Clip range for target policy smoothing noise
action_noise	Exploration noise during rollout collection

In [ ]:

model = TD3(
    "MlpPolicy",
    verbose=1,
    learning_rate=1e-3,
    buffer_size=100_000,
    learning_starts=10_000,
    batch_size=256,
    tau=0.005,
    gamma=0.99,
    gradient_steps=1,
    policy_delay=2,
    target_policy_noise=0.2,
    target_noise_clip=0.5,
    action_noise=action_noise,
    policy_kwargs={"net_arch": [256, 256]},
)
print(model.policy)

model = TD3( "MlpPolicy", verbose=1, learning_rate=1e-3, buffer_size=100_000, learning_starts=10_000, batch_size=256, tau=0.005, gamma=0.99, gradient_steps=1, policy_delay=2, target_policy_noise=0.2, target_noise_clip=0.5, action_noise=action_noise, policy_kwargs={"net_arch": [256, 256]}, ) print(model.policy)

Train¶

Train through the explicit RLTrainer entry point.

In [ ]:

trainer = RLTrainer(max_timesteps=50_000, enable_progress_bar=True)
trainer.fit(model, datamodule=RLEnvDataModule(train_env="LunarLanderContinuous-v2"))

trainer = RLTrainer(max_timesteps=50_000, enable_progress_bar=True) trainer.fit(model, datamodule=RLEnvDataModule(train_env="LunarLanderContinuous-v2"))

Evaluate¶

In [ ]:

eval_results = trainer.test(
    env=gym.make("LunarLanderContinuous-v2"),
    n_eval_episodes=10,
    verbose=False,
)
eval_results[0]

eval_results = trainer.test( env=gym.make("LunarLanderContinuous-v2"), n_eval_episodes=10, verbose=False, ) eval_results[0]