Model description:

Here we evaluate encoder-only part of the pretrained FRED-T5-1.7B model (https://russiansuperglue.com/login/submit_info/1936), resulting 760M parameters (2.2 times smaller). We fine-tune the model separately for each RussianSuperGLUE task. For LiDiRus we start with TERRa-finetuned model. For RWSD we use the majority class baseline. For each task we submit best-performing checkpoint (saving each epoch, but more frequently for RCB, PARus and RuCoS) based on validation metrics. No fixes were applied to the datasets. No filters/fixes were applied to datasets.

Parameter description:

Hyper-parameters for fine-tuning: batch size of 16, epochs {10, 20, 30}, lr {1e-06, 1e-04, 1e-5, 2e-5, 3e-5}, linear lr scheduler, warmup ratio {0.02, 0.05}, weight decay {0, 0.01, 0.1}.