Nov. 18, 2020, 3:36 p.m.
Team: SberDevices
Model url: https://huggingface.co/sberbank-ai/rugpt3small_based_on_gpt2
Dataset | Score | Metric |
---|---|---|
LiDiRus | -0.013 | Matthew`s Corr |
RCB | 0.356 / 0.473 | F1/Acc |
PARus | 0.562 | Accuracy |
MuSeRC | 0.653 / 0.221 | F1a/Em |
TERRa | 0.488 | Accuracy |
RUSSE | 0.57 | Accuracy |
RWSD | 0.669 | Accuracy |
DaNetQA | 0.61 | Accuracy |
RuCoS | 0.21 / 0.204 | F1/EM |
https://huggingface.co/sberbank-ai/rugpt3small_based_on_gpt2
standard
Category | Score |
---|---|
LOGIC | -0.0026653911281437344 |
KNOWLEDGE | 0.049106419683116166 |
PREDICATE-ARGUMENT STRUCTURE | 0.0 |
LEXICAL SEMANTICS | -0.05036130062634135 |
Lexical Semantics - Lexical Entailment | -0.1258455564269084 |
---|---|
Lexical Semantics - Morphological Negation | 0.0 |
Lexical Semantics - Factivity | 0.0 |
Lexical Semantics - Symmetry/Collectivity | 0.0 |
Lexical Semantics - Redundancy | 0.0 |
Lexical Semantics - Named Entities | 0.1690308509457033 |
Lexical Semantics - Quantifiers | 0.0 |
Predicate-Argument Structure Core Args | 0.0 |
Predicate-Argument Structure Prepositional Phrases | 0.0 |
Predicate-Argument Structure Ellipsis/Implicits | 0.0 |
Predicate-Argument Structure Anaphora/Coreference | 0.0 |
Predicate-Argument Structure Active/Passive | 0.0 |
Predicate-Argument Structure Nominalization | 0.0 |
Predicate-Argument Structure Genitives/Partitives | 0.0 |
Predicate-Argument Structure Datives | 0.0 |
Predicate-Argument Structure Relative Clauses | 0.0 |
Predicate-Argument Structure Coordination Scopes | 0.0 |
Predicate-Argument Structure Intersectivity | 0.0 |
Predicate-Argument Structure Restrictivity | 0.0 |
Logic Negation | 0.0 |
Logic Double Negation | 0.0 |
Logic Interval/Numbers | 0.0 |
Logic Conjuction | 0.0 |
Logic Disjunction | 0.0 |
Logic Conditionals | 0.0 |
Logic Universal | 0.0 |
Logic Existential | 0.0 |
Logic Temporal | -0.007053982594841415 |
Logic Upward Monotone | 0.0 |
Logic Downward Monotone | 0.0 |
Logic Non-Monotonic | 0.0 |
Knowledge Common Sense | 0.0 |
Knowledge World Knowledge | 0.04764016705577461 |
Dataset | Speed | RAM |
---|---|---|
LiDiRus | 176 | 2.36 |
RCB | 289 | 2.37 |
PARus | 872 | 2.36 |
MuSeRC | 4 | 2.38 |
TERRa | 319 | 2.37 |
RUSSE | 163 | 2.36 |
RWSD | 105 | 2.36 |
DaNetQA | 97 | 2.38 |
RuCoS | 8 | 2.38 |