Skip to content

PalaestrAI

flowcean.palaestrai is the facade for Soft Actor-Critic learning through PalaestrAI. It exports SACLearner and SACModel. Install the optional dependencies described in the installation guide before using this backend.

palaestrai

PalaestrAI Soft Actor-Critic learning facade.

Classes

SACLearner 

SACLearner(actuator_ids: list[str], sensor_ids: list[str], agent_objective: Objective, *, replay_size: int = int(1000000.0), fc_dims: Sequence[int] = (256, 256), activation: str = 'torch.nn.ReLU', gamma: float = 0.99, polyak: float = 0.995, lr: float = 0.001, batch_size: int = 100, update_after: int = 1000, update_every: int = 50)

Bases: ActiveLearner

Learner class for the palaestrAI SAC agent.

Initialize the SAC learner.

Parameters:

Name Type Description Default
actuator_ids list[str]

The IDs of actuators the learner should use to interact with the environment.

 required
sensor_ids list[str]

The IDs of sensors the learner should be able to see from the environment.

 required
agent_objective Objective

The objective function that takes environment rewards and converts them to an objective for the agent.

 required
replay_size int

Maximum length of replay buffer.

 int(1000000.0)
fc_dims Sequence[int]

Dimensions of the hidden layers of the agent's actor and critic networks. "fc" stands for "fully connected".

 (256, 256)
activation str

Activation function to use

 'torch.nn.ReLU'
gamma float

Discount factor. (Always between 0 and 1.)

 0.99
polyak float

Interpolation factor in polyak averaging for target networks. Target networks are updated towards main networks according to: $\(\theta_{\text{targ}} \leftarrow \rho \theta_{ \text{targ}} + (1-\rho) \theta,\)$ where \(\rho\) is polyak. (Always between 0 and 1, usually close to 1.)

 0.995
lr float

Learning rate (used for both policy and value learning).

 0.001
batch_size int

Minibatch size for SGD.

 100
update_after int

Number of env interactions to collect before starting to do gradient descent updates. Ensures replay buffer is full enough for useful updates.

 1000
update_every int

Number of env interactions that should elapse between gradient descent updates. Note: Regardless of how long you wait between updates, the ratio of environment interactions to gradient steps is locked to 1.

 50
Attributes
model instance-attribute 
model: SACModel
brain instance-attribute 
brain: SACBrain
agent_objective instance-attribute 
agent_objective: Objective = agent_objective
objectives instance-attribute 
objectives: list[float]
rewards instance-attribute 
rewards: list[list[ActiveInterface]] = []
actuator_ids instance-attribute 
actuator_ids: list[str] = actuator_ids
sensor_ids instance-attribute 
sensor_ids: list[str] = sensor_ids
action instance-attribute 
action: Action
observation instance-attribute 
observation: Observation
objective_values instance-attribute 
objective_values = []
brain_params instance-attribute 
brain_params = {'replay_size': replay_size, 'fc_dims': fc_dims, 'activation': activation, 'gamma': gamma, 'polyak': polyak, 'lr': lr, 'batch_size': batch_size, 'update_after': update_after, 'update_every': update_every}
Methods:
setup 
setup(action: Action, observation: Observation) -> None
learn_active 
learn_active(action: Action, observation: Observation) -> Model
propose_action 
propose_action(observation: Observation) -> Action
save 
save(file_path: str) -> None
load 
load(file_path: str) -> None

SACModel 

SACModel(action: Action, observation: Observation, sensor_ids: list[str], actuator_ids: list[str], model: Any, *, start_steps: int = 10000, training_mode: bool = False)

Bases: Model

Attributes
action instance-attribute 
action: Action = action
observation instance-attribute 
observation: Observation = observation
sensor_ids instance-attribute 
sensor_ids: list[str] = sensor_ids
actuator_ids instance-attribute 
actuator_ids: list[str] = actuator_ids
muscle instance-attribute 
muscle: SACMuscle = SACMuscle(start_steps)
data_for_brain instance-attribute 
data_for_brain: dict[str, Any] = {}
Methods:
predict 
predict(input_features: Observation) -> Action
save_state 
save_state() -> dict[str, Any]
load_from_state classmethod 
load_from_state(state: dict[str, Any]) -> Self
update 
update(update: Any) -> None
train 
train() -> None
eval 
eval() -> None