Submission RuBERT conversational

June 4, 2020, 3:26 p.m.

Team: AGI NLP

Model url: http://files.deeppavlov.ai/deeppavlov_data/bert/ru_conversational_cased_L-12_H-768_A-12.tar.gz


Total score: 0.546

Dataset Score Metric
LiDiRus 0.186 Matthew`s Corr
RCB 0.468 / 0.432 F1/Acc
PARus 0.61 Accuracy
MuSeRC 0.656 / 0.256 F1a/Em
TERRa 0.639 Accuracy
RUSSE 0.894 Accuracy
RWSD 0.675 Accuracy
DaNetQA 0.749 Accuracy
RuCoS 0.255 / 0.251 F1/EM
Model description:

Conversational RuBERT was trained by DeepPavlov on OpenSubtitles, Dirty, Pikabu, and Social Media segment of Taiga corpus. We assembled new vocabulary for Conversational RuBERT model on this data and initialized model with RuBERT. How we train model in jiant see in our repo, file: `Russian_SuperGLUE_example.ipynb`


Parameter description:

Conversational RuBERT was trained by DeepPavlov on OpenSubtitles, Dirty, Pikabu, and Social Media segment of Taiga corpus. We assembled new vocabulary for Conversational RuBERT model on this data and initialized model with RuBERT.

Diagnostic (Matthew`s Correlation): 0.186

Category Score
LOGIC 0.145576527967411
KNOWLEDGE 0.239252998092825
PREDICATE-ARGUMENT STRUCTURE 0.19405781006762
LEXICAL SEMANTICS 0.179254269922103
Lexical Semantics - Lexical Entailment 0.0427960492510913
Lexical Semantics - Morphological Negation 0.148607523353427
Lexical Semantics - Factivity 0.132453235706504
Lexical Semantics - Symmetry/Collectivity 0.0
Lexical Semantics - Redundancy -0.0601929265428846
Lexical Semantics - Named Entities 0.169030850945703
Lexical Semantics - Quantifiers 0.212508292628994
Predicate-Argument Structure Core Args 0.20839738967947
Predicate-Argument Structure Prepositional Phrases 0.295119372861731
Predicate-Argument Structure Ellipsis/Implicits 0.0587220219514703
Predicate-Argument Structure Anaphora/Coreference 0.256382281261804
Predicate-Argument Structure Active/Passive 0.0364565532716506
Predicate-Argument Structure Nominalization 0.188561808316413
Predicate-Argument Structure Genitives/Partitives 0.25
Predicate-Argument Structure Datives 0.490990253030983
Predicate-Argument Structure Relative Clauses 0.372104203767625
Predicate-Argument Structure Coordination Scopes 0.168345124585359
Predicate-Argument Structure Intersectivity 0.17884778999622
Predicate-Argument Structure Restrictivity -0.1455213750218
Logic Negation 0.018397408083176
Logic Double Negation 0.100503781525921
Logic Interval/Numbers 0.0699825065597668
Logic Conjuction 0.387298334620742
Logic Disjunction 0.0699825065597668
Logic Conditionals -0.1008712554448
Logic Universal 0.433200112721982
Logic Existential 0.244599795235114
Logic Temporal 0.129988774181506
Logic Upward Monotone 0.27640672769878
Logic Downward Monotone 0.00530140428942425
Logic Non-Monotonic 0.102443899885196
Knowledge Common Sense 0.182075291299721
Knowledge World Knowledge 0.284019661275609