13 сентября 2022 г. 8:59
Команда Yandex Research
Ссылка на модель https://huggingface.co/yandex/RuLeanALBERT
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,403 | Кор, коэффициент Мэтью |
| RCB | 0,361 / 0,413 | F1/Точность |
| PARus | 0,796 | Точность |
| MuSeRC | 0,874 / 0,654 | F1a/Em |
| TERRa | 0,812 | Точность |
| RUSSE | 0,789 | Точность |
| RWSD | 0,669 | Точность |
| DaNetQA | 0,76 | Точность |
| RuCoS | 0,9 / 0,902 | F1/EM |
RuLeanALBERT is a 2.9B Transformer encoder for Russian language trained on a mixture of the YaLM corpus, Taiga, RDT, and Wikipedia. It has several architectural differences from regular Transformers, namely weight sharing, rotary positional embeddings, GeGLU activations, and pre-normalization. Repo: https://github.com/yandex-research/RuLeanALBERT Habr: https://habr.com/ru/company/yandex/blog/688234/
| Категория | Результат |
|---|---|
| LOGIC | 0,25766842280097046 |
| KNOWLEDGE | 0,33438659081581373 |
| PREDICATE-ARGUMENT STRUCTURE | 0,3900261443729555 |
| LEXICAL SEMANTICS | 0,5105617927842396 |
| Lexical Semantics - Lexical Entailment | 0,45392064950160177 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,45175395145262565 |
| Lexical Semantics - Factivity | 0,43381600524612796 |
| Lexical Semantics - Symmetry/Collectivity | 0,43852900965351466 |
| Lexical Semantics - Redundancy | 0,5897678246195885 |
| Lexical Semantics - Named Entities | 0,5590169943749475 |
| Lexical Semantics - Quantifiers | 0,5348810406899015 |
| Predicate-Argument Structure Core Args | 0,5070925528371099 |
| Predicate-Argument Structure Prepositional Phrases | 0,6715144652340956 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,3263956049169334 |
| Predicate-Argument Structure Anaphora/Coreference | 0,15646620130993677 |
| Predicate-Argument Structure Active/Passive | 0,4083133966424866 |
| Predicate-Argument Structure Nominalization | 0,4445906192369001 |
| Predicate-Argument Structure Genitives/Partitives | 0,4900980294098034 |
| Predicate-Argument Structure Datives | 0,49099025303098287 |
| Predicate-Argument Structure Relative Clauses | 0,2537340189666186 |
| Predicate-Argument Structure Coordination Scopes | 0,42365927286816174 |
| Predicate-Argument Structure Intersectivity | 0,34502923976608185 |
| Predicate-Argument Structure Restrictivity | -0,014760248092334923 |
| Logic Negation | 0,11177050727031347 |
| Logic Double Negation | 0,20759971844307282 |
| Logic Interval/Numbers | 0,08877935232369505 |
| Logic Conjuction | 0,5270462766947299 |
| Logic Disjunction | 0,1396215401769611 |
| Logic Conditionals | 0,1972421118046462 |
| Logic Universal | 0,6700593942604899 |
| Logic Existential | 0,25677629550654774 |
| Logic Temporal | 0,10776235844876533 |
| Logic Upward Monotone | 0,6701754385964912 |
| Logic Downward Monotone | -0,3212669683257919 |
| Logic Non-Monotonic | 0,1286978904175574 |
| Knowledge Common Sense | 0,44831667596163405 |
| Knowledge World Knowledge | 0,20313571895099006 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |