RotRNN: Modelling Long Sequences with Rotations
Published in NGSM ICML Workshop, 2024
Linear recurrent models, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling benchmarks. Despite their success, they come with a number of drawbacks, most notably their complex initialisation and normalisation schemes. In this work, we address some of these issues by proposing RotRNN – a linear recurrent model which utilises the convenient properties of rotation matrices. We show that RotRNN provides a simple model with fewer theoretical assumptions than prior works, with a practical implementation that remains faithful to its theoretical derivation, achieving comparable scores to the LRU and SSMs on several long sequence modelling datasets.
Recommended citation: Dolga, Rares, et al. "RotRNN: Modelling Long Sequences with Rotations." arXiv preprint arXiv:2407.07239 (2024).
Download Paper