Submission RuGPT3Medium

Nov. 18, 2020, 3:34 p.m.

Team: sberdevices

Model url: https://huggingface.co/sberbank-ai/rugpt3medium_based_on_gpt2


Total score: 0.468

Dataset Score Metric
LiDiRus 0.01 Matthew`s Corr
RCB 0.372 / 0.461 F1/Acc
PARus 0.598 Accuracy
MuSeRC 0.706 / 0.308 F1a/Em
TERRa 0.505 Accuracy
RUSSE 0.642 Accuracy
RWSD 0.669 Accuracy
DaNetQA 0.634 Accuracy
RuCoS 0.23 / 0.224 F1/EM
Model description:

https://huggingface.co/sberbank-ai/rugpt3medium_based_on_gpt2


Parameter description:

https://huggingface.co/sberbank-ai/rugpt3medium_based_on_gpt2

Diagnostic (Matthew`s Correlation): 0.01

Category Score
LOGIC -0.03675581840635848
KNOWLEDGE 0.00884512993363901
PREDICATE-ARGUMENT STRUCTURE -0.021774965254905874
LEXICAL SEMANTICS 0.07626687869258576
Lexical Semantics - Lexical Entailment 0.07316844086241091
Lexical Semantics - Morphological Negation 0.2672612419124244
Lexical Semantics - Factivity -0.14433756729740643
Lexical Semantics - Symmetry/Collectivity 0.12171612389003691
Lexical Semantics - Redundancy -0.1444869078105018
Lexical Semantics - Named Entities 0.14907119849998599
Lexical Semantics - Quantifiers 0.03575901245155421
Predicate-Argument Structure Core Args -0.011111111111111112
Predicate-Argument Structure Prepositional Phrases 0.14173748623694876
Predicate-Argument Structure Ellipsis/Implicits 0.10213764617139019
Predicate-Argument Structure Anaphora/Coreference -0.13856455728036748
Predicate-Argument Structure Active/Passive -0.3059179896521843
Predicate-Argument Structure Nominalization -0.2581988897471611
Predicate-Argument Structure Genitives/Partitives -0.28867513459481287
Predicate-Argument Structure Datives -0.2182178902359924
Predicate-Argument Structure Relative Clauses 0.06666666666666667
Predicate-Argument Structure Coordination Scopes 0.10482848367219183
Predicate-Argument Structure Intersectivity -0.009250935259013151
Predicate-Argument Structure Restrictivity -0.09335200560186732
Logic Negation -0.00882398741607738
Logic Double Negation -0.2842676218074806
Logic Interval/Numbers 0.053875681986269665
Logic Conjuction 0.019518001458970664
Logic Disjunction 0.0
Logic Conditionals -0.29277002188455997
Logic Universal 0.08058229640253803
Logic Existential 0.15724272550828775
Logic Temporal -0.2735506022160966
Logic Upward Monotone 0.22710998958306755
Logic Downward Monotone 0.035955873264830976
Logic Non-Monotonic 0.06832298136645963
Knowledge Common Sense -0.04530785056456245
Knowledge World Knowledge 0.032122142172456535