Model description:

The underlying model (contains 280M parameters) was pre-trained by Microsoft on the CC100 multi-lingual dataset (Russian included). We further fine-tune the pre-trained model separately for each RussianSuperGLUE task. For LiDiRus we start with TERRa-finetuned model. For RWSD we use the majority class baseline. For each task we submit best-performing checkpoint (saving each epoch) based on validation metrics.

Parameter description:

Hyper-parameters for fine-tuning mdeberta-v3-base: batch size {8, 32}, epochs {20, 30}, lr {1e-05, 2e-05}, lr scheduler {constant, linear}, warmup ratio {0.02, 0.05, 0.1}, weight decay {0, 0.01, 0.1}.