Submission Plain RuBERT



Total score: 0.524

Dataset Score Metric
Diagnostic -0.026 Matthew's Corr
Russian CB 0.338/0.393 F1/Acc
PARus 0.532 Accuracy
MuSeRC 0.712/0.309 F1a/Em
TERRa 0.636 Accuracy
RUSSE 0.877 Accuracy
RWSD 0.662 Accuracy
DaNetQA 0.78 Accuracy
RuCoS 0.38/0.379 F1/EM
Model description:

RuBERT cased was trained on the Russian part of Wikipedia and news data. We used this training data to build vocabulary of Russian subtokens and took multilingual version of BERT-base as initialization for RuBERT


Parameter description:

RuBERT cased was trained on the Russian part of Wikipedia and news data. We used this training data to build vocabulary of Russian subtokens and took multilingual version of BERT-base as initialization for RuBERT


Diagnostic (Matthew`s Correlation): -0.026

Category Score
LOGIC -0.0662526431795982
KNOWLEDGE -0.01781375112362455
PREDICATE-ARGUMENT STRUCTURE -0.02742685942168112
LEXICAL SEMANTICS 0.06261045121383906
Lexical Semantics - Lexical Entailment 0.20005879542992555
Lexical Semantics - Morphological Negation -0.31180478223116176
Lexical Semantics - Factivity 0.13969968905677815
Lexical Semantics - Symmetry/Collectivity 0.22821773229381923
Lexical Semantics - Redundancy -0.6756639246921762
Lexical Semantics - Named Entities 0.10050378152592121
Lexical Semantics - Quantifiers -0.16824404674701446
Predicate-Argument Structure Core Args 0.23696305584563251
Predicate-Argument Structure Prepositional Phrases -0.15394498874964566
Predicate-Argument Structure Ellipsis/Implicits 0.1846372364689991
Predicate-Argument Structure Anaphora/Coreference -0.016540792339471234
Predicate-Argument Structure Active/Passive 0.0
Predicate-Argument Structure Nominalization -0.12171612389003691
Predicate-Argument Structure Genitives/Partitives 0.0
Predicate-Argument Structure Datives -0.0657951694959769
Predicate-Argument Structure Relative Clauses -0.2
Predicate-Argument Structure Coordination Scopes -0.1188643277625491
Predicate-Argument Structure Intersectivity -0.07509392614826384
Predicate-Argument Structure Restrictivity -0.09335200560186732
Logic Negation 0.05704384514499037
Logic Double Negation -0.21102672654165874
Logic Interval/Numbers -0.1630980846658687
Logic Conjuction -0.2817180849095055
Logic Disjunction 0.032892970457377636
Logic Conditionals 0.009523809523809525
Logic Universal -0.3567530340063379
Logic Existential -0.36689969285267143
Logic Temporal -0.12998877418150556
Logic Upward Monotone -0.10054155780296499
Logic Downward Monotone -0.08597285067873303
Logic Non-Monotonic 0.16850509274845385
Knowledge Common Sense 0.03300552979416381
Knowledge World Knowledge -0.07190869663379389