Submission Multilingual BERT



Total score: 0.542

Dataset Score Metric
Diagnostic 0.157 Matthew's Corr
Russian CB 0.365/0.425 F1/Acc
PARus 0.588 Accuracy
MuSeRC 0.626/0.253 F1a/Em
TERRa 0.62 Accuracy
RUSSE 0.84 Accuracy
RWSD 0.675 Accuracy
DaNetQA 0.79 Accuracy
RuCoS 0.371/0.367 F1/EM
Model description:

MultilingualBERT, version "bert-base-multilingual-cased" in transformers


Parameter description:

Pre-trained for every task with early stopping and limited number of epochs


Diagnostic (Matthew`s Correlation): 0.157

Category Score
LOGIC 0.09964181678655076
KNOWLEDGE 0.1093699904861369
PREDICATE-ARGUMENT STRUCTURE 0.16931338548666588
LEXICAL SEMANTICS 0.1920486019231863
Lexical Semantics - Lexical Entailment 0.019713064408633168
Lexical Semantics - Morphological Negation 0.1543033499620919
Lexical Semantics - Factivity 0.2602896031476768
Lexical Semantics - Symmetry/Collectivity -0.12171612389003691
Lexical Semantics - Redundancy 0.6756639246921762
Lexical Semantics - Named Entities 0.055641488407465724
Lexical Semantics - Quantifiers 0.2597177743262246
Predicate-Argument Structure Core Args 0.297107355542109
Predicate-Argument Structure Prepositional Phrases 0.34665482822302573
Predicate-Argument Structure Ellipsis/Implicits 0.1613743060919757
Predicate-Argument Structure Anaphora/Coreference -0.02667200160053352
Predicate-Argument Structure Active/Passive 0.3244428422615251
Predicate-Argument Structure Nominalization 0.2363163359313514
Predicate-Argument Structure Genitives/Partitives 0.375
Predicate-Argument Structure Datives 0.28511240114923325
Predicate-Argument Structure Relative Clauses -0.06666666666666667
Predicate-Argument Structure Coordination Scopes 0.16834512458535864
Predicate-Argument Structure Intersectivity 0.19379591094941798
Predicate-Argument Structure Restrictivity 0.2748737083745107
Logic Negation -0.03445364007928464
Logic Double Negation -0.07537783614444091
Logic Interval/Numbers 0.010300524052492094
Logic Conjuction 0.3253956867279843
Logic Disjunction -0.0528680532637681
Logic Conditionals -0.10776235844876533
Logic Universal 0.6446583712203042
Logic Existential 0.2516299559794226
Logic Temporal 0.06210273191490665
Logic Upward Monotone 0.3612343522752406
Logic Downward Monotone 0.269069117598525
Logic Non-Monotonic -0.16850509274845385
Knowledge Common Sense 0.023943707502436275
Knowledge World Knowledge 0.1866948732559256