Submission ruT5-large finetune

Sept. 20, 2021, 6:57 a.m.

Team: Sberdevices

Model url: https://huggingface.co/sberbank-ai/ruT5-large


Total score: 0.686

Dataset Score Metric
LiDiRus 0.32 Matthew`s Corr
RCB 0.45 / 0.532 F1/Acc
PARus 0.764 Accuracy
MuSeRC 0.855 / 0.608 F1a/Em
TERRa 0.775 Accuracy
RUSSE 0.773 Accuracy
RWSD 0.669 Accuracy
DaNetQA 0.79 Accuracy
RuCoS 0.86 / 0.859 F1/EM
Model description:

Russian T5-large model. 740 million parameters, trained on 300G texts of different domains.


Parameter description:

-

Diagnostic (Matthew`s Correlation): 0.32

Category Score
LOGIC 0.20694602549177668
KNOWLEDGE 0.2617642051708644
PREDICATE-ARGUMENT STRUCTURE 0.3005668454878599
LEXICAL SEMANTICS 0.41830327755878044
Lexical Semantics - Lexical Entailment 0.3727642044390084
Lexical Semantics - Morphological Negation 0.6172133998483676
Lexical Semantics - Factivity 0.2721655269759087
Lexical Semantics - Symmetry/Collectivity 0.43852900965351466
Lexical Semantics - Redundancy 0.5247497678328021
Lexical Semantics - Named Entities 0.40482045237636816
Lexical Semantics - Quantifiers 0.37854910518078255
Predicate-Argument Structure Core Args 0.47544348941583686
Predicate-Argument Structure Prepositional Phrases 0.4454180314547771
Predicate-Argument Structure Ellipsis/Implicits 0.1649915822768611
Predicate-Argument Structure Anaphora/Coreference 0.15646620130993677
Predicate-Argument Structure Active/Passive 0.23461612570453771
Predicate-Argument Structure Nominalization 0.3373495424699933
Predicate-Argument Structure Genitives/Partitives 0.15724272550828775
Predicate-Argument Structure Datives 0.43463356032809364
Predicate-Argument Structure Relative Clauses 0.29814239699997197
Predicate-Argument Structure Coordination Scopes 0.33454829277463405
Predicate-Argument Structure Intersectivity 0.3107299650387684
Predicate-Argument Structure Restrictivity 0.09335200560186732
Logic Negation -0.09103804161885289
Logic Double Negation 0.3892494720807615
Logic Interval/Numbers 0.12121248708414586
Logic Conjuction 0.38666666666666666
Logic Disjunction 0.20430156738209967
Logic Conditionals -0.050964719143762556
Logic Universal 0.4029114820126901
Logic Existential 0.3851644432598216
Logic Temporal 0.11396057645963795
Logic Upward Monotone 0.6701754385964912
Logic Downward Monotone -0.24715576637149037
Logic Non-Monotonic 0.15445574836957118
Knowledge Common Sense 0.31542785259678374
Knowledge World Knowledge 0.1866948732559256

Performance:

Dataset Speed RAM
LiDiRus - -
RCB - -
PARus - -
MuSeRC - -
TERRa - -
RUSSE - -
RWSD - -
DaNetQA - -
RuCoS - -