Submission ruT5-base-finetune

June 23, 2021, 10:34 a.m.

Team: SberDevices

Model url: https://huggingface.co/sberbank-ai/ruT5-base


Total score: 0.596

Dataset Score Metric
LiDiRus 0.267 Matthew`s Corr
RCB 0.307 / 0.468 F1/Acc
PARus 0.554 Accuracy
MuSeRC 0.769 / 0.446 F1a/Em
TERRa 0.692 Accuracy
RUSSE 0.682 Accuracy
RWSD 0.669 Accuracy
DaNetQA 0.732 Accuracy
RuCoS 0.79 / 0.752 F1/EM
Model description:

Russian T5-base model. 220 million parameters, trained on 300G texts of different domains.


Parameter description:

-

Diagnostic (Matthew`s Correlation): 0.267

Category Score
LOGIC 0.15622738039722758
KNOWLEDGE 0.21448722544758325
PREDICATE-ARGUMENT STRUCTURE 0.32003228881838025
LEXICAL SEMANTICS 0.31403620034011387
Lexical Semantics - Lexical Entailment 0.14169809491720062
Lexical Semantics - Morphological Negation 0.3003124875651863
Lexical Semantics - Factivity 0.4486515850484713
Lexical Semantics - Symmetry/Collectivity -0.03227486121839514
Lexical Semantics - Redundancy 0.3448281541042982
Lexical Semantics - Named Entities 0.3333333333333333
Lexical Semantics - Quantifiers 0.20730678680184425
Predicate-Argument Structure Core Args 0.29514591494904874
Predicate-Argument Structure Prepositional Phrases 0.593992399790426
Predicate-Argument Structure Ellipsis/Implicits 0.3263956049169334
Predicate-Argument Structure Anaphora/Coreference 0.20833333333333334
Predicate-Argument Structure Active/Passive 0.2135744251723958
Predicate-Argument Structure Nominalization 0.023570226039551584
Predicate-Argument Structure Genitives/Partitives 0.4900980294098034
Predicate-Argument Structure Datives 0.5238095238095238
Predicate-Argument Structure Relative Clauses 0.4472135954999579
Predicate-Argument Structure Coordination Scopes 0.20001902768532515
Predicate-Argument Structure Intersectivity 0.23234812346321748
Predicate-Argument Structure Restrictivity 0.24331548436407657
Logic Negation -0.0391953200678408
Logic Double Negation 0.06364909472854591
Logic Interval/Numbers 0.015982110049512804
Logic Conjuction 0.4601321676666517
Logic Disjunction 0.1191759143062248
Logic Conditionals 0.1111111111111111
Logic Universal 0.2987012987012987
Logic Existential 0.38981938376529196
Logic Temporal -0.15244937348544793
Logic Upward Monotone 0.44353708548308607
Logic Downward Monotone -0.10984700727621793
Logic Non-Monotonic -0.03217447260438935
Knowledge Common Sense 0.17471452631765566
Knowledge World Knowledge 0.23308947827422227

Performance:

Dataset Speed RAM
LiDiRus - -
RCB - -
PARus - -
MuSeRC - -
TERRa - -
RUSSE - -
RWSD - -
DaNetQA - -
RuCoS - -