Submission Slavic BERT



Total score: 0.483

Dataset Score Metric
Diagnostic -0.01 Matthew's Corr
Russian CB 0.34/0.418 F1/Acc
PARus 0.526 Accuracy
MuSeRC 0.677/0.27 F1a/Em
TERRa 0.566 Accuracy
RUSSE 0.829 Accuracy
RWSD 0.578 Accuracy
DaNetQA 0.759 Accuracy
RuCoS 0.24/0.242 F1/EM
Model description:

SlavicBERT was trained on Russian News and four Wikipedias: Bulgarian, Czech, Polish, and Russian. Subtoken vocabulary was built using this data. Multilingual BERT was used as an initialization for SlavicBERT. The model is described in our ACL paper 2


Parameter description:

SlavicBERT was trained on Russian News and four Wikipedias: Bulgarian, Czech, Polish, and Russian. Subtoken vocabulary was built using this data. Multilingual BERT was used as an initialization for SlavicBERT. The model is described in our ACL paper 2


Diagnostic (Matthew`s Correlation): -0.01

Category Score
LOGIC -0.06323105515352936
KNOWLEDGE -0.045173275975792876
PREDICATE-ARGUMENT STRUCTURE 0.019678502385693086
LEXICAL SEMANTICS -0.061984954906365526
Lexical Semantics - Lexical Entailment 0.03785959930369332
Lexical Semantics - Morphological Negation -0.39009474880274697
Lexical Semantics - Factivity -0.1653953539259914
Lexical Semantics - Symmetry/Collectivity 0.29488391230979427
Lexical Semantics - Redundancy -0.27348301713730944
Lexical Semantics - Named Entities -0.1690308509457033
Lexical Semantics - Quantifiers -0.012515654358043972
Predicate-Argument Structure Core Args 0.28888888888888886
Predicate-Argument Structure Prepositional Phrases -0.09079139649239867
Predicate-Argument Structure Ellipsis/Implicits 0.0
Predicate-Argument Structure Anaphora/Coreference -0.20380437270105392
Predicate-Argument Structure Active/Passive 0.2724804837691825
Predicate-Argument Structure Nominalization 0.20672455764868078
Predicate-Argument Structure Genitives/Partitives 0.11470786693528087
Predicate-Argument Structure Datives 0.15018785229652767
Predicate-Argument Structure Relative Clauses -0.09759000729485333
Predicate-Argument Structure Coordination Scopes -0.1175019408963036
Predicate-Argument Structure Intersectivity 0.031453034622341385
Predicate-Argument Structure Restrictivity 0.0
Logic Negation -0.019853063847782893
Logic Double Negation 0.1005037815259212
Logic Interval/Numbers 0.14589603651080282
Logic Conjuction 0.0
Logic Disjunction 0.0
Logic Conditionals 0.0
Logic Universal -0.19347649748797374
Logic Existential -0.24459979523511427
Logic Temporal -0.1868706368604627
Logic Upward Monotone 0.029617443887954616
Logic Downward Monotone 0.052768795428464264
Logic Non-Monotonic 0.0
Knowledge Common Sense -0.04007862820909901
Knowledge World Knowledge -0.06381919253395192