Submission RuGPT3Large

Nov. 18, 2020, 3:33 p.m.

Team: sberdevices

Model url: https://huggingface.co/sberbank-ai/rugpt3large_based_on_gpt2


Total score: 0.505

Dataset Score Metric
LiDiRus 0.231 Matthew`s Corr
RCB 0.417 / 0.484 F1/Acc
PARus 0.584 Accuracy
MuSeRC 0.729 / 0.333 F1a/Em
TERRa 0.654 Accuracy
RUSSE 0.647 Accuracy
RWSD 0.636 Accuracy
DaNetQA 0.604 Accuracy
RuCoS 0.21 / 0.202 F1/EM
Model description:

https://huggingface.co/sberbank-ai/rugpt3large_based_on_gpt2


Parameter description:

standard

Diagnostic (Matthew`s Correlation): 0.231

Category Score
LOGIC 0.11933663119357156
KNOWLEDGE 0.20781115309502002
PREDICATE-ARGUMENT STRUCTURE 0.2596971971125482
LEXICAL SEMANTICS 0.22258462187108333
Lexical Semantics - Lexical Entailment 0.08036864720442753
Lexical Semantics - Morphological Negation -0.08333333333333333
Lexical Semantics - Factivity 0.3567530340063379
Lexical Semantics - Symmetry/Collectivity 0.125
Lexical Semantics - Redundancy 0.5247497678328021
Lexical Semantics - Named Entities 0.22360679774997896
Lexical Semantics - Quantifiers 0.3577734237456226
Predicate-Argument Structure Core Args 0.3516258877670335
Predicate-Argument Structure Prepositional Phrases 0.5474265370777632
Predicate-Argument Structure Ellipsis/Implicits 0.04286204165613664
Predicate-Argument Structure Anaphora/Coreference 0.22264349695528307
Predicate-Argument Structure Active/Passive 0.08962646244131739
Predicate-Argument Structure Nominalization 0.3522819383711917
Predicate-Argument Structure Genitives/Partitives 0.050251890762960605
Predicate-Argument Structure Datives 0.28511240114923325
Predicate-Argument Structure Relative Clauses 0.33954987505086615
Predicate-Argument Structure Coordination Scopes 0.366007208697342
Predicate-Argument Structure Intersectivity 0.18492720525389514
Predicate-Argument Structure Restrictivity 0.08947924869885988
Logic Negation -0.06341828617898013
Logic Double Negation -0.07537783614444091
Logic Interval/Numbers -0.19235526336800596
Logic Conjuction 0.5270462766947299
Logic Disjunction 0.03134361106013412
Logic Conditionals 0.07100716024967263
Logic Universal 0.6700593942604899
Logic Existential 0.4530333378893934
Logic Temporal 0.06210273191490665
Logic Upward Monotone 0.5322656777660987
Logic Downward Monotone -0.43938802910487174
Logic Non-Monotonic 0.07881104062391008
Knowledge Common Sense 0.30212645553936207
Knowledge World Knowledge 0.10622304182533429