Новый сабмит RuBERT conversational

4 июня 2020 г. 15:26

Команда AGI NLP

Ссылка на модель http://files.deeppavlov.ai/deeppavlov_data/bert/ru_conversational_cased_L-12_H-768_A-12.tar.gz

Результат бейзлайна: 0,546

Датасет Результат Метрика
LiDiRus 0,186 Кор, коэффициент Мэтью
RCB 0,468 / 0,432 F1/Точность
PARus 0,61 Точность
MuSeRC 0,656 / 0,256 F1a/Em
TERRa 0,639 Точность
RUSSE 0,894 Точность
RWSD 0,675 Точность
DaNetQA 0,749 Точность
RuCoS 0,255 / 0,251 F1/EM
Описание модели:

Conversational RuBERT was trained by DeepPavlov on OpenSubtitles, Dirty, Pikabu, and Social Media segment of Taiga corpus. We assembled new vocabulary for Conversational RuBERT model on this data and initialized model with RuBERT. How we train model in jiant see in our repo, file: `Russian_SuperGLUE_example.ipynb`


Описание параметров:

Conversational RuBERT was trained by DeepPavlov on OpenSubtitles, Dirty, Pikabu, and Social Media segment of Taiga corpus. We assembled new vocabulary for Conversational RuBERT model on this data and initialized model with RuBERT.

Диагностика: 0,186

Категория Результат
LOGIC 0,145576527967411
KNOWLEDGE 0,239252998092825
PREDICATE-ARGUMENT STRUCTURE 0,19405781006762
LEXICAL SEMANTICS 0,179254269922103
Lexical Semantics - Lexical Entailment 0,0427960492510913
Lexical Semantics - Morphological Negation 0,148607523353427
Lexical Semantics - Factivity 0,132453235706504
Lexical Semantics - Symmetry/Collectivity 0,0
Lexical Semantics - Redundancy -0,0601929265428846
Lexical Semantics - Named Entities 0,169030850945703
Lexical Semantics - Quantifiers 0,212508292628994
Predicate-Argument Structure Core Args 0,20839738967947
Predicate-Argument Structure Prepositional Phrases 0,295119372861731
Predicate-Argument Structure Ellipsis/Implicits 0,0587220219514703
Predicate-Argument Structure Anaphora/Coreference 0,256382281261804
Predicate-Argument Structure Active/Passive 0,0364565532716506
Predicate-Argument Structure Nominalization 0,188561808316413
Predicate-Argument Structure Genitives/Partitives 0,25
Predicate-Argument Structure Datives 0,490990253030983
Predicate-Argument Structure Relative Clauses 0,372104203767625
Predicate-Argument Structure Coordination Scopes 0,168345124585359
Predicate-Argument Structure Intersectivity 0,17884778999622
Predicate-Argument Structure Restrictivity -0,1455213750218
Logic Negation 0,018397408083176
Logic Double Negation 0,100503781525921
Logic Interval/Numbers 0,0699825065597668
Logic Conjuction 0,387298334620742
Logic Disjunction 0,0699825065597668
Logic Conditionals -0,1008712554448
Logic Universal 0,433200112721982
Logic Existential 0,244599795235114
Logic Temporal 0,129988774181506
Logic Upward Monotone 0,27640672769878
Logic Downward Monotone 0,00530140428942425
Logic Non-Monotonic 0,102443899885196
Knowledge Common Sense 0,182075291299721
Knowledge World Knowledge 0,284019661275609