Results

Best robust algorithm vs best standard algorithm

In each setting, online and offline, the two algorithms are selected by overall performance: among the robust and among the standard algorithms of the setting, we take the one with the highest score under shift, averaged over all tasks and shift conditions of the library-wide evaluation. Online, this selects DR-SAC from the 6 robust algorithms and SAC from the 2 standard algorithms. Offline, it selects ATLA-IQL from the 8 robust algorithms and SynthER from the 4 standard algorithms.

Normalized score of the best robust and the best standard algorithm across the severity quartiles: DR-SAC and SAC online, ATLA-IQL and SynthER offline.
Figure 1. Normalized score of the best robust and the best standard algorithm across the severity quartiles Q1–Q4. Left: online, DR-SAC and SAC. Right: offline, ATLA-IQL and SynthER. The labels give the gap at Q1 and Q4.

Online, the best robust algorithm stays above the best standard algorithm at every severity level. The gap is +14.1 at Q1 and +5.2 at Q4. Over all levels DR-SAC scores 66.8 and SAC 56.7.

Offline, the best robust algorithm also stays above the best standard algorithm at every severity level. The gap is +7.1 at Q1 and +1.9 at Q4. Over all levels ATLA-IQL scores 56.9 and SynthER 53.2.