| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.096 | Matthew`s Corr |
| RCB | 0.302 / 0.418 | F1/Acc |
| PARus | 0.676 | Accuracy |
| MuSeRC | 0.74 / 0.546 | F1a/Em |
| TERRa | 0.573 | Accuracy |
| RUSSE | 0.565 | Accuracy |
| RWSD | 0.649 | Accuracy |
| DaNetQA | 0.59 | Accuracy |
| RuCoS | 0.67 / 0.665 | F1/EM |
This submission was made by Few-Shot methods only ruGPT-3 Xl - 1.3 billion parameters All tasks solved by ranging the perplexity of the answer variants Checkpoints and documentation can be found here: https://github.com/sberbank-ai/ru-gpts https://github.com/sberbank-ai/ru-gpts/tree/master/gw
| Category | Score |
|---|---|
| LOGIC | -0.007250629987786855 |
| KNOWLEDGE | 0.059773386339694506 |
| PREDICATE-ARGUMENT STRUCTURE | 0.13997903121271693 |
| LEXICAL SEMANTICS | 0.18737835348183993 |
| Lexical Semantics - Lexical Entailment | 0.029950995334911953 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.0 |
| Lexical Semantics - Factivity | 0.06900655593423542 |
| Lexical Semantics - Symmetry/Collectivity | 0.2921186973360886 |
| Lexical Semantics - Redundancy | 0.45652173913043476 |
| Lexical Semantics - Named Entities | 0.24253562503633297 |
| Lexical Semantics - Quantifiers | 0.20823613034492347 |
| Predicate-Argument Structure Core Args | 0.18433668044583404 |
| Predicate-Argument Structure Prepositional Phrases | 0.23123776351036351 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.08554415042912275 |
| Predicate-Argument Structure Anaphora/Coreference | -0.01692566771665739 |
| Predicate-Argument Structure Active/Passive | 0.05353033790313108 |
| Predicate-Argument Structure Nominalization | 0.0 |
| Predicate-Argument Structure Genitives/Partitives | -0.25 |
| Predicate-Argument Structure Datives | 0.3563483225498992 |
| Predicate-Argument Structure Relative Clauses | 0.09759000729485333 |
| Predicate-Argument Structure Coordination Scopes | 0.35186577527449836 |
| Predicate-Argument Structure Intersectivity | 0.12100224017315647 |
| Predicate-Argument Structure Restrictivity | -0.1455213750217998 |
| Logic Negation | 0.0399615133948266 |
| Logic Double Negation | 0.21102672654165874 |
| Logic Interval/Numbers | -0.25225301366160086 |
| Logic Conjuction | -0.1084652289093281 |
| Logic Disjunction | 0.15291057030852148 |
| Logic Conditionals | -0.14285714285714285 |
| Logic Universal | 0.2548235957188128 |
| Logic Existential | -0.15724272550828775 |
| Logic Temporal | -0.13261933174731558 |
| Logic Upward Monotone | 0.10054155780296499 |
| Logic Downward Monotone | 0.03008284187980934 |
| Logic Non-Monotonic | -0.34151450937027694 |
| Knowledge Common Sense | -0.03115501519756839 |
| Knowledge World Knowledge | 0.14184850482497574 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |