Submission RuBERT conversational

Nov. 18, 2020, 3:31 p.m.

Team: DeepPavlov

Model url: http://files.deeppavlov.ai/deeppavlov_data/bert/ru_conversational_cased_L-12_H-768_A-12_pt.tar.gz

Total score: 0.5

Dataset Score Metric
LiDiRus 0.178 Matthew`s Corr
RCB 0.452 / 0.484 F1/Acc
PARus 0.508 Accuracy
MuSeRC 0.687 / 0.278 F1a/Em
TERRa 0.64 Accuracy
RUSSE 0.729 Accuracy
RWSD 0.669 Accuracy
DaNetQA 0.606 Accuracy
RuCoS 0.22 / 0.218 F1/EM
Model description:

Conversational RuBERT was trained by DeepPavlov on OpenSubtitles, Dirty, Pikabu, and Social Media segment of Taiga corpus. We assembled new vocabulary for Conversational RuBERT model on this data and initialized model with RuBERT. How we train model in jiant see in our repo, file: `Russian_SuperGLUE_example.ipynb`


Parameter description:

Conversational RuBERT was trained by DeepPavlov on OpenSubtitles, Dirty, Pikabu, and Social Media segment of Taiga corpus. We assembled new vocabulary for Conversational RuBERT model on this data and initialized model with RuBERT.

Diagnostic (Matthew`s Correlation): 0.178

Category Score
LOGIC 0.08909703425730169
KNOWLEDGE 0.22685372713785582
PREDICATE-ARGUMENT STRUCTURE 0.18755413627356057
LEXICAL SEMANTICS 0.17880052424736642
Lexical Semantics - Lexical Entailment 0.08997060091656868
Lexical Semantics - Morphological Negation 0.042257712736425826
Lexical Semantics - Factivity 0.1908790423082963
Lexical Semantics - Symmetry/Collectivity 0.0
Lexical Semantics - Redundancy 0.45652173913043476
Lexical Semantics - Named Entities 0.07018624063435965
Lexical Semantics - Quantifiers 0.016787316051946218
Predicate-Argument Structure Core Args 0.2141057445312242
Predicate-Argument Structure Prepositional Phrases 0.23271817721614832
Predicate-Argument Structure Ellipsis/Implicits 0.014265349750363764
Predicate-Argument Structure Anaphora/Coreference 0.2960528777048222
Predicate-Argument Structure Active/Passive -0.003948992518393949
Predicate-Argument Structure Nominalization 0.28499398669032056
Predicate-Argument Structure Genitives/Partitives 0.375
Predicate-Argument Structure Datives 0.3361227822436775
Predicate-Argument Structure Relative Clauses 0.19740669742132735
Predicate-Argument Structure Coordination Scopes 0.10482848367219183
Predicate-Argument Structure Intersectivity 0.07984231865671275
Predicate-Argument Structure Restrictivity 0.09335200560186732
Logic Negation -0.03223161048648345
Logic Double Negation 0.07537783614444091
Logic Interval/Numbers 0.06998250655976682
Logic Conjuction 0.24809590313546123
Logic Disjunction -0.16447838793172298
Logic Conditionals -0.14285714285714285
Logic Universal 0.16116459280507606
Logic Existential 0.36689969285267143
Logic Temporal 0.26590801173915524
Logic Upward Monotone 0.2135744251723958
Logic Downward Monotone -0.08112739143148366
Logic Non-Monotonic 0.18389242812245682
Knowledge Common Sense 0.22216215518070082
Knowledge World Knowledge 0.21289279690944252

Performance:

Dataset Speed RAM
LiDiRus 171 2.39
RCB 289 2.39
PARus 718 2.39
MuSeRC 4 2.40
TERRa 302 2.39
RUSSE 255 2.39
RWSD 101 2.39
DaNetQA 103 2.40
RuCoS 8 2.40