Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning

Abstract

In this paper, we experiment with novelty-based variants of OpenAI-ES, NS-ES and NSR-ES algorithms, and evaluate their effectiveness in training complex transformer-based architectures designed for the problem of reinforcement learning, such as Decision Transformers. We also test whether we can accelerate the novelty-based training of these larger models by seeding the training with a pretrained model. The experimental results were mixed. NS-ES showed progress, but it would clearly need many more iterations for it to yield interesting results. NSR-ES, on the other hand, proved quite capable of being straightforwardly used on larger models, since its performance appears as similar between the feed-forward model and Decision Transformer, as it is for the baseline objective-based OpenAI-ES.