Submission RuGPT3Small

Nov. 18, 2020, 3:36 p.m.

Team: SberDevices

Model url: https://huggingface.co/sberbank-ai/rugpt3small_based_on_gpt2


Total score: 0.438

Dataset Score Metric
LiDiRus -0.013 Matthew`s Corr
RCB 0.356 / 0.473 F1/Acc
PARus 0.562 Accuracy
MuSeRC 0.653 / 0.221 F1a/Em
TERRa 0.488 Accuracy
RUSSE 0.57 Accuracy
RWSD 0.669 Accuracy
DaNetQA 0.61 Accuracy
RuCoS 0.21 / 0.204 F1/EM
Model description:

https://huggingface.co/sberbank-ai/rugpt3small_based_on_gpt2


Parameter description:

standard

Diagnostic (Matthew`s Correlation): -0.013

Category Score
LOGIC -0.0026653911281437344
KNOWLEDGE 0.049106419683116166
PREDICATE-ARGUMENT STRUCTURE 0.0
LEXICAL SEMANTICS -0.05036130062634135
Lexical Semantics - Lexical Entailment -0.1258455564269084
Lexical Semantics - Morphological Negation 0.0
Lexical Semantics - Factivity 0.0
Lexical Semantics - Symmetry/Collectivity 0.0
Lexical Semantics - Redundancy 0.0
Lexical Semantics - Named Entities 0.1690308509457033
Lexical Semantics - Quantifiers 0.0
Predicate-Argument Structure Core Args 0.0
Predicate-Argument Structure Prepositional Phrases 0.0
Predicate-Argument Structure Ellipsis/Implicits 0.0
Predicate-Argument Structure Anaphora/Coreference 0.0
Predicate-Argument Structure Active/Passive 0.0
Predicate-Argument Structure Nominalization 0.0
Predicate-Argument Structure Genitives/Partitives 0.0
Predicate-Argument Structure Datives 0.0
Predicate-Argument Structure Relative Clauses 0.0
Predicate-Argument Structure Coordination Scopes 0.0
Predicate-Argument Structure Intersectivity 0.0
Predicate-Argument Structure Restrictivity 0.0
Logic Negation 0.0
Logic Double Negation 0.0
Logic Interval/Numbers 0.0
Logic Conjuction 0.0
Logic Disjunction 0.0
Logic Conditionals 0.0
Logic Universal 0.0
Logic Existential 0.0
Logic Temporal -0.007053982594841415
Logic Upward Monotone 0.0
Logic Downward Monotone 0.0
Logic Non-Monotonic 0.0
Knowledge Common Sense 0.0
Knowledge World Knowledge 0.04764016705577461

Performance:

Dataset Speed RAM
LiDiRus 176 2.36
RCB 289 2.37
PARus 872 2.36
MuSeRC 4 2.38
TERRa 319 2.37
RUSSE 163 2.36
RWSD 105 2.36
DaNetQA 97 2.38
RuCoS 8 2.38