Results

Gain over the base algorithm and over standard RL pool

Ten robust methods share a base algorithm (PPO, SAC or IQL) with a standard RL reference. On the library-wide grid of 220 configurations, each of them is compared with its own base algorithm, which shows what the mechanism adds, and with the standard RL pool of its regime, which also contains the strength of the base learner.

Pooled standard and robust scores across the severity quartiles, and the gain of ten robust methods over their base algorithm and over the standard RL pool, online and offline.
Figure 1. Left: pooled Standard and Robust scores across the severity quartiles Q1–Q4. Right: the gain of each method over its own base algorithm (left group of each block) and over the mean of the standard RL references of the same regime (right group), online and offline.

Online, the gain comes from the mechanism. ATLA-PPO, ATLA-SA-PPO and DR-SAC gain +7.2, +13.4 and +10.1 over their base algorithm. Online mechanisms train in the environment, so a policy that fails under a perturbation is penalized by its true return and the generated experience is real.

Offline, the gain comes mostly from the base learner. The five IQL-based methods are +10.1 to +14.0 above the offline standard RL pool but at most +3.8 above IQL. Offline mechanisms must generate such experience from a fixed dataset with a learned model, with no environment feedback to correct it. Additionally, offline algorithms update conservatively to limit value overestimation, which constrains the gain any mechanism can add.