18 ноября 2020 г. 15:30
Команда DeepPavlov
Ссылка на модель http://files.deeppavlov.ai/deeppavlov_data/bert/rubert_cased_L-12_H-768_A-12_pt.tar.gz
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,191 | Кор, коэффициент Мэтью |
| RCB | 0,367 / 0,463 | F1/Точность |
| PARus | 0,574 | Точность |
| MuSeRC | 0,711 / 0,324 | F1a/Em |
| TERRa | 0,642 | Точность |
| RUSSE | 0,726 | Точность |
| RWSD | 0,669 | Точность |
| DaNetQA | 0,639 | Точность |
| RuCoS | 0,32 / 0,314 | F1/EM |
RuBERT cased was trained on the Russian part of Wikipedia and news data. We used this training data to build vocabulary of Russian subtokens and took multilingual version of BERT-base as initialization for RuBERT
RuBERT cased was trained on the Russian part of Wikipedia and news data. We used this training data to build vocabulary of Russian subtokens and took multilingual version of BERT-base as initialization for RuBERT
| Категория | Результат |
|---|---|
| LOGIC | 0,12307487469477564 |
| KNOWLEDGE | 0,19866500041629373 |
| PREDICATE-ARGUMENT STRUCTURE | 0,21191399021657556 |
| LEXICAL SEMANTICS | 0,21690049170508532 |
| Lexical Semantics - Lexical Entailment | 0,17118419700436516 |
|---|---|
| Lexical Semantics - Morphological Negation | -0,2672612419124244 |
| Lexical Semantics - Factivity | 0,21320071635561044 |
| Lexical Semantics - Symmetry/Collectivity | 0,0 |
| Lexical Semantics - Redundancy | 0,45652173913043476 |
| Lexical Semantics - Named Entities | 0,24253562503633297 |
| Lexical Semantics - Quantifiers | 0,10042420607280685 |
| Predicate-Argument Structure Core Args | 0,2571428571428571 |
| Predicate-Argument Structure Prepositional Phrases | 0,3488716748619316 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,04286204165613664 |
| Predicate-Argument Structure Anaphora/Coreference | 0,31126486012524596 |
| Predicate-Argument Structure Active/Passive | 0,27640672769878033 |
| Predicate-Argument Structure Nominalization | 0,2363163359313514 |
| Predicate-Argument Structure Genitives/Partitives | -0,11470786693528087 |
| Predicate-Argument Structure Datives | 0,5238095238095238 |
| Predicate-Argument Structure Relative Clauses | 0,24913643956121992 |
| Predicate-Argument Structure Coordination Scopes | 0,16146816171752817 |
| Predicate-Argument Structure Intersectivity | 0,03545133231866123 |
| Predicate-Argument Structure Restrictivity | 0,09335200560186732 |
| Logic Negation | 0,01027493670658483 |
| Logic Double Negation | 0,055048188256318034 |
| Logic Interval/Numbers | 0,08260937904561348 |
| Logic Conjuction | 0,3279566366999691 |
| Logic Disjunction | 0,06998250655976682 |
| Logic Conditionals | 0,03253000243161777 |
| Logic Universal | 0,5229975846277625 |
| Logic Existential | 0,014678923792502562 |
| Logic Temporal | 0,0 |
| Logic Upward Monotone | 0,27640672769878033 |
| Logic Downward Monotone | -0,1504142093990467 |
| Logic Non-Monotonic | -0,07881104062391008 |
| Knowledge Common Sense | 0,22530395136778114 |
| Knowledge World Knowledge | 0,14184850482497574 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | 165 | 2.39 |
| RCB | 295 | 2.39 |
| PARus | 1070 | 2.39 |
| MuSeRC | 4 | 2.40 |
| TERRa | 297 | 2.39 |
| RUSSE | 226 | 2.39 |
| RWSD | 102 | 2.39 |
| DaNetQA | 118 | 2.40 |
| RuCoS | 9 | 2.40 |