29 мая 2021 г. 6:54
Команда SberDevices
Ссылка на модель https://huggingface.co/sberbank-ai/sbert_large_mt_nlu_ru
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,218 | Кор, коэффициент Мэтью |
| RCB | 0,351 / 0,486 | F1/Точность |
| PARus | 0,498 | Точность |
| MuSeRC | 0,642 / 0,319 | F1a/Em |
| TERRa | 0,637 | Точность |
| RUSSE | 0,657 | Точность |
| RWSD | 0,675 | Точность |
| DaNetQA | 0,697 | Точность |
| RuCoS | 0,35 / 0,347 | F1/EM |
BERT large model multitask (cased) for Sentence Embeddings in Russian language. The model was fine-tuned on NER, toxic, Dialogue and RussianSuperGLUE tasks, code written on tensorflow. For better quality we use for sentence representation mean token embeddings.
| Категория | Результат |
|---|---|
| LOGIC | 0,16192611130344906 |
| KNOWLEDGE | 0,22740828237325783 |
| PREDICATE-ARGUMENT STRUCTURE | 0,20139781205785764 |
| LEXICAL SEMANTICS | 0,2813434604687791 |
| Lexical Semantics - Lexical Entailment | 0,2148928384990026 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,2828894749305018 |
| Lexical Semantics - Factivity | 0,1781741612749496 |
| Lexical Semantics - Symmetry/Collectivity | 0,22821773229381923 |
| Lexical Semantics - Redundancy | 0,30692490702242564 |
| Lexical Semantics - Named Entities | 0,16692446522239718 |
| Lexical Semantics - Quantifiers | 0,47387910220727386 |
| Predicate-Argument Structure Core Args | 0,3304857994026801 |
| Predicate-Argument Structure Prepositional Phrases | 0,26354595321588503 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,23570226039551584 |
| Predicate-Argument Structure Anaphora/Coreference | -0,002651969513727742 |
| Predicate-Argument Structure Active/Passive | 0,1382147381437882 |
| Predicate-Argument Structure Nominalization | 0,24552983583446694 |
| Predicate-Argument Structure Genitives/Partitives | 0,10482848367219183 |
| Predicate-Argument Structure Datives | 0,3563483225498992 |
| Predicate-Argument Structure Relative Clauses | 0,01642880193633814 |
| Predicate-Argument Structure Coordination Scopes | 0,17910620335162064 |
| Predicate-Argument Structure Intersectivity | 0,4129955231527934 |
| Predicate-Argument Structure Restrictivity | -0,1895424836601307 |
| Logic Negation | 0,09642916726065977 |
| Logic Double Negation | 0,2727272727272727 |
| Logic Interval/Numbers | -0,019157088122605363 |
| Logic Conjuction | 0,10540925533894599 |
| Logic Disjunction | 0,1286978904175574 |
| Logic Conditionals | 0,14285714285714285 |
| Logic Universal | 0,26856632724128343 |
| Logic Existential | 0,060522753266880246 |
| Logic Temporal | 0,12434118282549846 |
| Logic Upward Monotone | 0,3398363161942986 |
| Logic Downward Monotone | 0,10984700727621793 |
| Logic Non-Monotonic | -0,2163856337073152 |
| Knowledge Common Sense | 0,2514207350602513 |
| Knowledge World Knowledge | 0,1854598701600781 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |