Nov. 18, 2020, 3:33 p.m.
Team: SberDevices
Model url: https://huggingface.co/sberbank-ai/rugpt3large_based_on_gpt2
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.231 | Matthew`s Corr |
| RCB | 0.417 / 0.484 | F1/Acc |
| PARus | 0.584 | Accuracy |
| MuSeRC | 0.729 / 0.333 | F1a/Em |
| TERRa | 0.654 | Accuracy |
| RUSSE | 0.647 | Accuracy |
| RWSD | 0.636 | Accuracy |
| DaNetQA | 0.604 | Accuracy |
| RuCoS | 0.21 / 0.202 | F1/EM |
https://huggingface.co/sberbank-ai/rugpt3large_based_on_gpt2
standard
| Category | Score |
|---|---|
| LOGIC | 0.11933663119357156 |
| KNOWLEDGE | 0.20781115309502002 |
| PREDICATE-ARGUMENT STRUCTURE | 0.2596971971125482 |
| LEXICAL SEMANTICS | 0.22258462187108333 |
| Lexical Semantics - Lexical Entailment | 0.08036864720442753 |
|---|---|
| Lexical Semantics - Morphological Negation | -0.08333333333333333 |
| Lexical Semantics - Factivity | 0.3567530340063379 |
| Lexical Semantics - Symmetry/Collectivity | 0.125 |
| Lexical Semantics - Redundancy | 0.5247497678328021 |
| Lexical Semantics - Named Entities | 0.22360679774997896 |
| Lexical Semantics - Quantifiers | 0.3577734237456226 |
| Predicate-Argument Structure Core Args | 0.3516258877670335 |
| Predicate-Argument Structure Prepositional Phrases | 0.5474265370777632 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.04286204165613664 |
| Predicate-Argument Structure Anaphora/Coreference | 0.22264349695528307 |
| Predicate-Argument Structure Active/Passive | 0.08962646244131739 |
| Predicate-Argument Structure Nominalization | 0.3522819383711917 |
| Predicate-Argument Structure Genitives/Partitives | 0.050251890762960605 |
| Predicate-Argument Structure Datives | 0.28511240114923325 |
| Predicate-Argument Structure Relative Clauses | 0.33954987505086615 |
| Predicate-Argument Structure Coordination Scopes | 0.366007208697342 |
| Predicate-Argument Structure Intersectivity | 0.18492720525389514 |
| Predicate-Argument Structure Restrictivity | 0.08947924869885988 |
| Logic Negation | -0.06341828617898013 |
| Logic Double Negation | -0.07537783614444091 |
| Logic Interval/Numbers | -0.19235526336800596 |
| Logic Conjuction | 0.5270462766947299 |
| Logic Disjunction | 0.03134361106013412 |
| Logic Conditionals | 0.07100716024967263 |
| Logic Universal | 0.6700593942604899 |
| Logic Existential | 0.4530333378893934 |
| Logic Temporal | 0.06210273191490665 |
| Logic Upward Monotone | 0.5322656777660987 |
| Logic Downward Monotone | -0.43938802910487174 |
| Logic Non-Monotonic | 0.07881104062391008 |
| Knowledge Common Sense | 0.30212645553936207 |
| Knowledge World Knowledge | 0.10622304182533429 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | 69 | 7.50 |
| RCB | 53 | 7.50 |
| PARus | 137 | 7.50 |
| MuSeRC | 1 | 7.49 |
| TERRa | 61 | 7.50 |
| RUSSE | 75 | 7.49 |
| RWSD | 49 | 7.51 |
| DaNetQA | 27 | 7.49 |
| RuCoS | 2 | 7.49 |