Aug. 18, 2021, 11:49 a.m.
Team: SberDevices
Model url: https://huggingface.co/sberbank-ai/ruRoberta-large
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.343 | Matthew`s Corr |
| RCB | 0.357 / 0.518 | F1/Acc |
| PARus | 0.722 | Accuracy |
| MuSeRC | 0.861 / 0.63 | F1a/Em |
| TERRa | 0.801 | Accuracy |
| RUSSE | 0.748 | Accuracy |
| RWSD | 0.669 | Accuracy |
| DaNetQA | 0.82 | Accuracy |
| RuCoS | 0.87 / 0.867 | F1/EM |
Russian RoBERTa large model. 355ml parameters, trained in 250 GB Russian texts.
| Category | Score |
|---|---|
| LOGIC | 0.1820012297621496 |
| KNOWLEDGE | 0.2698822742714764 |
| PREDICATE-ARGUMENT STRUCTURE | 0.34132316093258036 |
| LEXICAL SEMANTICS | 0.489878955484335 |
| Lexical Semantics - Lexical Entailment | 0.4238498361791447 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.6366624736381951 |
| Lexical Semantics - Factivity | 0.289805474447035 |
| Lexical Semantics - Symmetry/Collectivity | 0.5477225575051661 |
| Lexical Semantics - Redundancy | 0.7985494095046906 |
| Lexical Semantics - Named Entities | 0.5427204202399745 |
| Lexical Semantics - Quantifiers | 0.35043832202523123 |
| Predicate-Argument Structure Core Args | 0.5111111111111111 |
| Predicate-Argument Structure Prepositional Phrases | 0.3972493441698867 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.24433144650348906 |
| Predicate-Argument Structure Anaphora/Coreference | 0.07955908541183225 |
| Predicate-Argument Structure Active/Passive | 0.44353708548308607 |
| Predicate-Argument Structure Nominalization | 0.6024640760767093 |
| Predicate-Argument Structure Genitives/Partitives | 0.6875 |
| Predicate-Argument Structure Datives | 0.7637626158259734 |
| Predicate-Argument Structure Relative Clauses | 0.29277002188455997 |
| Predicate-Argument Structure Coordination Scopes | 0.2516299559794226 |
| Predicate-Argument Structure Intersectivity | 0.20679318146353196 |
| Predicate-Argument Structure Restrictivity | 0.09335200560186732 |
| Logic Negation | 0.11166573401651324 |
| Logic Double Negation | 0.4271410714851651 |
| Logic Interval/Numbers | -0.12121248708414586 |
| Logic Conjuction | 0.24575816200113812 |
| Logic Disjunction | -0.09647574015265818 |
| Logic Conditionals | 0.02486823656509969 |
| Logic Universal | 0.5606119105813882 |
| Logic Existential | 0.42365927286816174 |
| Logic Temporal | -0.030303030303030304 |
| Logic Upward Monotone | 0.3612343522752406 |
| Logic Downward Monotone | 0.06726727939963124 |
| Logic Non-Monotonic | 0.2163856337073152 |
| Knowledge Common Sense | 0.21361974637528325 |
| Knowledge World Knowledge | 0.3084094312371165 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |