Model description:

The underlying model (contains 560M parameters) was pre-trained by Facebook on the CC100 multi-lingual dataset (Russian included). We further fine-tune the pre-trained model separately for each RussianSuperGLUE task. For LiDiRus we start with TERRa-finetuned model. For RWSD we use the majority class baseline. For each task we submit best-performing checkpoint (saving each epoch) based on validation metrics.

Parameter description:

Hyper-parameters for fine-tuning xlm-roberta-large: batch size {8, 16, 32, 64}, epochs {20, 30}, lr {1e-06, 2e-06, 1e-05, 3e-05}, lr scheduler {constant, linear}, warmup ratio {0.02, 0.05, 0.1}, weight decay {0, 0.01, 0.1}.