Submission RuBERT plain

Nov. 18, 2020, 3:30 p.m.

Team: DeepPavlov

Model url: http://files.deeppavlov.ai/deeppavlov_data/bert/rubert_cased_L-12_H-768_A-12_pt.tar.gz

Total score: 0.521

Dataset Score Metric
LiDiRus 0.191 Matthew`s Corr
RCB 0.367 / 0.463 F1/Acc
PARus 0.574 Accuracy
MuSeRC 0.711 / 0.324 F1a/Em
TERRa 0.642 Accuracy
RUSSE 0.726 Accuracy
RWSD 0.669 Accuracy
DaNetQA 0.639 Accuracy
RuCoS 0.32 / 0.314 F1/EM
Model description:

RuBERT cased was trained on the Russian part of Wikipedia and news data. We used this training data to build vocabulary of Russian subtokens and took multilingual version of BERT-base as initialization for RuBERT


Parameter description:

RuBERT cased was trained on the Russian part of Wikipedia and news data. We used this training data to build vocabulary of Russian subtokens and took multilingual version of BERT-base as initialization for RuBERT

Diagnostic (Matthew`s Correlation): 0.191

Category Score
LOGIC 0.12307487469477564
KNOWLEDGE 0.19866500041629373
PREDICATE-ARGUMENT STRUCTURE 0.21191399021657556
LEXICAL SEMANTICS 0.21690049170508532
Lexical Semantics - Lexical Entailment 0.17118419700436516
Lexical Semantics - Morphological Negation -0.2672612419124244
Lexical Semantics - Factivity 0.21320071635561044
Lexical Semantics - Symmetry/Collectivity 0.0
Lexical Semantics - Redundancy 0.45652173913043476
Lexical Semantics - Named Entities 0.24253562503633297
Lexical Semantics - Quantifiers 0.10042420607280685
Predicate-Argument Structure Core Args 0.2571428571428571
Predicate-Argument Structure Prepositional Phrases 0.3488716748619316
Predicate-Argument Structure Ellipsis/Implicits 0.04286204165613664
Predicate-Argument Structure Anaphora/Coreference 0.31126486012524596
Predicate-Argument Structure Active/Passive 0.27640672769878033
Predicate-Argument Structure Nominalization 0.2363163359313514
Predicate-Argument Structure Genitives/Partitives -0.11470786693528087
Predicate-Argument Structure Datives 0.5238095238095238
Predicate-Argument Structure Relative Clauses 0.24913643956121992
Predicate-Argument Structure Coordination Scopes 0.16146816171752817
Predicate-Argument Structure Intersectivity 0.03545133231866123
Predicate-Argument Structure Restrictivity 0.09335200560186732
Logic Negation 0.01027493670658483
Logic Double Negation 0.055048188256318034
Logic Interval/Numbers 0.08260937904561348
Logic Conjuction 0.3279566366999691
Logic Disjunction 0.06998250655976682
Logic Conditionals 0.03253000243161777
Logic Universal 0.5229975846277625
Logic Existential 0.014678923792502562
Logic Temporal 0.0
Logic Upward Monotone 0.27640672769878033
Logic Downward Monotone -0.1504142093990467
Logic Non-Monotonic -0.07881104062391008
Knowledge Common Sense 0.22530395136778114
Knowledge World Knowledge 0.14184850482497574

Performance:

Dataset Speed RAM
LiDiRus 165 2.39
RCB 295 2.39
PARus 1070 2.39
MuSeRC 4 2.40
TERRa 297 2.39
RUSSE 226 2.39
RWSD 102 2.39
DaNetQA 118 2.40
RuCoS 9 2.40