Nov. 18, 2020, 3:34 p.m.
Team: SberDevices
Model url: https://huggingface.co/sberbank-ai/rugpt3medium_based_on_gpt2
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.01 | Matthew`s Corr |
| RCB | 0.372 / 0.461 | F1/Acc |
| PARus | 0.598 | Accuracy |
| MuSeRC | 0.706 / 0.308 | F1a/Em |
| TERRa | 0.505 | Accuracy |
| RUSSE | 0.642 | Accuracy |
| RWSD | 0.669 | Accuracy |
| DaNetQA | 0.634 | Accuracy |
| RuCoS | 0.23 / 0.224 | F1/EM |
https://huggingface.co/sberbank-ai/rugpt3medium_based_on_gpt2
https://huggingface.co/sberbank-ai/rugpt3medium_based_on_gpt2
| Category | Score |
|---|---|
| LOGIC | -0.03675581840635848 |
| KNOWLEDGE | 0.00884512993363901 |
| PREDICATE-ARGUMENT STRUCTURE | -0.021774965254905874 |
| LEXICAL SEMANTICS | 0.07626687869258576 |
| Lexical Semantics - Lexical Entailment | 0.07316844086241091 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.2672612419124244 |
| Lexical Semantics - Factivity | -0.14433756729740643 |
| Lexical Semantics - Symmetry/Collectivity | 0.12171612389003691 |
| Lexical Semantics - Redundancy | -0.1444869078105018 |
| Lexical Semantics - Named Entities | 0.14907119849998599 |
| Lexical Semantics - Quantifiers | 0.03575901245155421 |
| Predicate-Argument Structure Core Args | -0.011111111111111112 |
| Predicate-Argument Structure Prepositional Phrases | 0.14173748623694876 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.10213764617139019 |
| Predicate-Argument Structure Anaphora/Coreference | -0.13856455728036748 |
| Predicate-Argument Structure Active/Passive | -0.3059179896521843 |
| Predicate-Argument Structure Nominalization | -0.2581988897471611 |
| Predicate-Argument Structure Genitives/Partitives | -0.28867513459481287 |
| Predicate-Argument Structure Datives | -0.2182178902359924 |
| Predicate-Argument Structure Relative Clauses | 0.06666666666666667 |
| Predicate-Argument Structure Coordination Scopes | 0.10482848367219183 |
| Predicate-Argument Structure Intersectivity | -0.009250935259013151 |
| Predicate-Argument Structure Restrictivity | -0.09335200560186732 |
| Logic Negation | -0.00882398741607738 |
| Logic Double Negation | -0.2842676218074806 |
| Logic Interval/Numbers | 0.053875681986269665 |
| Logic Conjuction | 0.019518001458970664 |
| Logic Disjunction | 0.0 |
| Logic Conditionals | -0.29277002188455997 |
| Logic Universal | 0.08058229640253803 |
| Logic Existential | 0.15724272550828775 |
| Logic Temporal | -0.2735506022160966 |
| Logic Upward Monotone | 0.22710998958306755 |
| Logic Downward Monotone | 0.035955873264830976 |
| Logic Non-Monotonic | 0.06832298136645963 |
| Knowledge Common Sense | -0.04530785056456245 |
| Knowledge World Knowledge | 0.032122142172456535 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | 106 | 4.39 |
| RCB | 102 | 4.39 |
| PARus | 270 | 4.39 |
| MuSeRC | 2 | 4.38 |
| TERRa | 111 | 4.39 |
| RUSSE | 106 | 4.38 |
| RWSD | 70 | 4.41 |
| DaNetQA | 45 | 4.41 |
| RuCoS | 3 | 4.38 |