Submission RuBERT conversational



Total score: 0.546

Dataset Score Metric
Diagnostic 0.186 Matthew's Corr
Russian CB 0.432/0.468 F1/Acc
PARus 0.61 Accuracy
MuSeRC 0.656/0.256 F1a/Em
TERRa 0.639 Accuracy
RUSSE 0.894 Accuracy
RWSD 0.675 Accuracy
DaNetQA 0.749 Accuracy
RuCoS 0.255/0.251 F1/EM
Model description:

Conversational RuBERT was trained by DeepPavlov on OpenSubtitles, Dirty, Pikabu, and Social Media segment of Taiga corpus. We assembled new vocabulary for Conversational RuBERT model on this data and initialized model with RuBERT. How we train model in jiant see in our repo, file: "Russian_SuperGLUE_example.ipynb"


Parameter description:

Conversational RuBERT was trained by DeepPavlov on OpenSubtitles, Dirty, Pikabu, and Social Media segment of Taiga corpus. We assembled new vocabulary for Conversational RuBERT model on this data and initialized model with RuBERT.


Diagnostic (Matthew`s Correlation): 0.186

Category Score
LOGIC 0.14557652796741136
KNOWLEDGE 0.23925299809282538
PREDICATE-ARGUMENT STRUCTURE 0.1940578100676201
LEXICAL SEMANTICS 0.1792542699221029
Lexical Semantics - Lexical Entailment 0.04279604925109129
Lexical Semantics - Morphological Negation 0.14860752335342742
Lexical Semantics - Factivity 0.13245323570650439
Lexical Semantics - Symmetry/Collectivity 0.0
Lexical Semantics - Redundancy -0.0601929265428846
Lexical Semantics - Named Entities 0.1690308509457033
Lexical Semantics - Quantifiers 0.21250829262899418
Predicate-Argument Structure Core Args 0.2083973896794696
Predicate-Argument Structure Prepositional Phrases 0.29511937286173096
Predicate-Argument Structure Ellipsis/Implicits 0.058722021951470346
Predicate-Argument Structure Anaphora/Coreference 0.2563822812618041
Predicate-Argument Structure Active/Passive 0.036456553271650594
Predicate-Argument Structure Nominalization 0.18856180831641267
Predicate-Argument Structure Genitives/Partitives 0.25
Predicate-Argument Structure Datives 0.49099025303098287
Predicate-Argument Structure Relative Clauses 0.3721042037676254
Predicate-Argument Structure Coordination Scopes 0.16834512458535864
Predicate-Argument Structure Intersectivity 0.17884778999621992
Predicate-Argument Structure Restrictivity -0.1455213750217998
Logic Negation 0.018397408083176008
Logic Double Negation 0.10050378152592121
Logic Interval/Numbers 0.06998250655976682
Logic Conjuction 0.3872983346207417
Logic Disjunction 0.06998250655976682
Logic Conditionals -0.10087125544480041
Logic Universal 0.4332001127219817
Logic Existential 0.24459979523511427
Logic Temporal 0.12998877418150556
Logic Upward Monotone 0.27640672769878033
Logic Downward Monotone 0.005301404289424247
Logic Non-Monotonic 0.10244389988519648
Knowledge Common Sense 0.1820752912997206
Knowledge World Knowledge 0.28401966127560896