Welcome to the Russian SuperGLUE benchmark

Modern universal language models and transformers such as BERT, ELMo, XLNet, RoBERTa and others need to be properly compared and evaluated. In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks.

We offer testing methodology based on tasks, typically proposed for “strong AI” — logic, commonsense, reasoning. Adhering to the GLUE and SuperGLUE methodology, we present a set of test tasks for general language understanding and leaderboard models.

For the first time a complete test for Russian language was developed, which is similar to its English analog. Many datasets were composed for the first time, and a leaderboard of models for the Russian language with comparable results is also presented.

Our talks and publications


Our Team


Tatiana Shavrina


Tatiana Shavrina is a PhD student at the Higher School of Economics and is the Head of RnD Department in NLP at Sberbank. Tatiana has presented her main works in the field of web сorpus сonstruction and universal transformers. Tatiana has also organized evaluation tracks for Russian in spelling correction, morphology, full UD annotation, ellypsis resolution.


Alena Fenogenova


Alena Fenogenova has a Master degree in Computational linguistics at the Higher School of Economics, Moscow. She has a number of publications in NLP field. Now Alena works in Sberbank in NLP research team. Her research interests are language understanding, Question Answering, Argument Mining, etc.


Valentin Malykh


Valentin Malykh has written his PhD thesis at Moscow Institute of Physics and Technology and defended it at Institute for Systems Programming, Russian Academy of Sciences in 2019. Dr. Malykh has more than 20 papers in NLP field, including publications on such conferences as NeurIPS, ACL, WSDM. Now Valentin is employed as a senior research scientist at Huawei Noah`s Ark laboratory.


Ekaterina Artemova


Ekaterina Artemova (née Chernyak) has received PhD from the Higher School of Economics, where she currently holds a PostDoc position and leads a newly established research group in natural language processing.


Anton Emelianov


Anton Emelyanov writes his PhD thesis at Moscow Institute of Physics and Technology. Now He is a member of the Laboratory for Computational Pragmatics. Anton is also employed as a research scientist at Sberbank, RnD in NLP team. His interests are language understanding, language models and generation.


Vladislav Mikhailov


Vladislav Mikhailov has a Master degree in Computational linguistics at the Higher School of Economics, Moscow. He works as an NLP Data Scientist at Sberbank. His main research interests are probing of language models and machine reading comprehension.


Maria Tikhonova


Maria Tikhonova is a PhD student in Computer Science at the Higher School of Economics, Moscow. She received a Masters’s degree in fundamental mechanics and mathematics at the Moscow State University. She works in Sberbank in the Research&Development in NLP department. Her interests are NLP and topic modeling.


Denis Shevelev


Denis Shevelev is a graduate of the Sholokhov State University (Moscow) and has two higher educations in the field of philology and journalism. Since 2005, works in copywriting and editing. His interests are NLU, sentiment analysis, and general intelligence. He is also the author of two graphic novels.


Taisiya Glushkova


Research intern at Higher School of Economics / Moscow, Russia
Taisiya has a Master degree in Computer Science at the Higher School of Economics, where she currently holds a Research Intern position in the research group in natural language processing. Previously worked at Unbabel, Diginetica and Sberbank.


Andrey Evlampiev


Andrey Evlampiev works at Sberbank as data science manager, focused on metrics and data engineering processes. Andrey studied at Moscow State University at Computer Science faculty, also has finished Yandex School of Data Analysis.