Oct. 16, 2023, 11:41 a.m.
Team: SberDevices
Model url: https://huggingface.co/ai-forever/ruElectra-large
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.197 | Matthew`s Corr |
| RCB | 0.386 / 0.459 | F1/Acc |
| PARus | 0.644 | Accuracy |
| MuSeRC | 0.549 / 0.078 | F1a/Em |
| TERRa | 0.583 | Accuracy |
| RUSSE | 0.632 | Accuracy |
| RWSD | 0.669 | Accuracy |
| DaNetQA | 0.627 | Accuracy |
| RuCoS | 0.61 / 0.607 | F1/EM |
ruElectra-large model of critic encoder class. It has 24 layers and hidden size 1024. It was trained on a Russian language corpus (100GB). Wordpiece tokenizer. Source data for training: Taiga, Lenta, OpenSubtitles, Wiki, etc. - all with ru lang. domain. Scoring pipeline: https://github.com/ai-forever/rsg-baselines
| Category | Score |
|---|---|
| LOGIC | 0.10344882418344328 |
| KNOWLEDGE | 0.18446096974096243 |
| PREDICATE-ARGUMENT STRUCTURE | 0.2395746447136374 |
| LEXICAL SEMANTICS | 0.23707730280981554 |
| Lexical Semantics - Lexical Entailment | 0.1448057711943421 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.21957751641341997 |
| Lexical Semantics - Factivity | 0.16012815380508713 |
| Lexical Semantics - Symmetry/Collectivity | 0.09128709291752768 |
| Lexical Semantics - Redundancy | 0.0 |
| Lexical Semantics - Named Entities | 0.24806946917841693 |
| Lexical Semantics - Quantifiers | 0.2140767569329586 |
| Predicate-Argument Structure Core Args | 0.15052428411666496 |
| Predicate-Argument Structure Prepositional Phrases | 0.3224469744632013 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.24514516892273003 |
| Predicate-Argument Structure Anaphora/Coreference | 0.19308582308458774 |
| Predicate-Argument Structure Active/Passive | 0.24433144650348906 |
| Predicate-Argument Structure Nominalization | 0.12171612389003691 |
| Predicate-Argument Structure Genitives/Partitives | 0.2182178902359924 |
| Predicate-Argument Structure Datives | 0.2182178902359924 |
| Predicate-Argument Structure Relative Clauses | 0.2277100170213244 |
| Predicate-Argument Structure Coordination Scopes | 0.3144854510165755 |
| Predicate-Argument Structure Intersectivity | 0.13429295209341927 |
| Predicate-Argument Structure Restrictivity | 0.17712297710801908 |
| Logic Negation | -0.17437145811572893 |
| Logic Double Negation | 0.45226701686664544 |
| Logic Interval/Numbers | -0.040522044923655395 |
| Logic Conjuction | 0.15491933384829668 |
| Logic Disjunction | 0.07460921467460303 |
| Logic Conditionals | 0.0849411985729376 |
| Logic Universal | 0.3959441875175622 |
| Logic Existential | -0.022875450543583874 |
| Logic Temporal | -0.34188172937891387 |
| Logic Upward Monotone | 0.1604088343952626 |
| Logic Downward Monotone | 0.049773755656108594 |
| Logic Non-Monotonic | 0.2163856337073152 |
| Knowledge Common Sense | 0.13232776605838764 |
| Knowledge World Knowledge | 0.20845916940990206 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |