Submission MT5 Large

Feb. 10, 2021, 11:29 a.m.

Team: AGI NLP

Model url: https://huggingface.co/google/mt5-large


Total score: 0.528

Dataset Score Metric
LiDiRus 0.061 Matthew`s Corr
RCB 0.366 / 0.454 F1/Acc
PARus 0.504 Accuracy
MuSeRC 0.844 / 0.543 F1a/Em
TERRa 0.561 Accuracy
RUSSE 0.633 Accuracy
RWSD 0.669 Accuracy
DaNetQA 0.657 Accuracy
RuCoS 0.57 / 0.562 F1/EM
Model description:

Multilingual T5 model fine-tuned on Russian SuperGLUE


Parameter description:

standard 10 epochs lr = 1e-5

Diagnostic (Matthew`s Correlation): 0.061

Category Score
LOGIC 0.056542787805431136
KNOWLEDGE 0.09735865003651223
PREDICATE-ARGUMENT STRUCTURE -0.0006367805561385829
LEXICAL SEMANTICS 0.1353825902684706
Lexical Semantics - Lexical Entailment 0.04279604925109129
Lexical Semantics - Morphological Negation 0.1355261854357877
Lexical Semantics - Factivity 0.1908790423082963
Lexical Semantics - Symmetry/Collectivity -0.2581988897471611
Lexical Semantics - Redundancy -0.08695652173913043
Lexical Semantics - Named Entities 0.0
Lexical Semantics - Quantifiers 0.19507587228904755
Predicate-Argument Structure Core Args -0.1539753448429908
Predicate-Argument Structure Prepositional Phrases 0.03192285726057405
Predicate-Argument Structure Ellipsis/Implicits 0.08554415042912275
Predicate-Argument Structure Anaphora/Coreference 0.03150630189063022
Predicate-Argument Structure Active/Passive -0.029617443887954616
Predicate-Argument Structure Nominalization 0.12171612389003691
Predicate-Argument Structure Genitives/Partitives -0.15309310892394865
Predicate-Argument Structure Datives 0.0
Predicate-Argument Structure Relative Clauses -0.06666666666666667
Predicate-Argument Structure Coordination Scopes 0.2773500981126146
Predicate-Argument Structure Intersectivity 0.15193428136569098
Predicate-Argument Structure Restrictivity -0.0771118547590483
Logic Negation -0.08269194339289726
Logic Double Negation 0.035533452725935076
Logic Interval/Numbers 0.1910789265895446
Logic Conjuction 0.18077538151554684
Logic Disjunction -0.2621269092188047
Logic Conditionals -0.14285714285714285
Logic Universal 0.025482359571881278
Logic Existential 0.36689969285267143
Logic Temporal -0.32328707534629597
Logic Upward Monotone 0.3612343522752406
Logic Downward Monotone 0.04756514941544941
Logic Non-Monotonic 0.14744195615489714
Knowledge Common Sense 0.17137570097266552
Knowledge World Knowledge 0.013856004982729668

Performance:

Dataset Speed RAM
LiDiRus - -
RCB - -
PARus - -
MuSeRC - -
TERRa - -
RUSSE - -
RWSD - -
DaNetQA - -
RuCoS - -