May 29, 2021, 6:54 a.m.
Team: SberDevices
Model url: https://huggingface.co/sberbank-ai/sbert_large_mt_nlu_ru
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.218 | Matthew`s Corr |
| RCB | 0.351 / 0.486 | F1/Acc |
| PARus | 0.498 | Accuracy |
| MuSeRC | 0.642 / 0.319 | F1a/Em |
| TERRa | 0.637 | Accuracy |
| RUSSE | 0.657 | Accuracy |
| RWSD | 0.675 | Accuracy |
| DaNetQA | 0.697 | Accuracy |
| RuCoS | 0.35 / 0.347 | F1/EM |
BERT large model multitask (cased) for Sentence Embeddings in Russian language. The model was fine-tuned on NER, toxic, Dialogue and RussianSuperGLUE tasks, code written on tensorflow. For better quality we use for sentence representation mean token embeddings.
| Category | Score |
|---|---|
| LOGIC | 0.16192611130344906 |
| KNOWLEDGE | 0.22740828237325783 |
| PREDICATE-ARGUMENT STRUCTURE | 0.20139781205785764 |
| LEXICAL SEMANTICS | 0.2813434604687791 |
| Lexical Semantics - Lexical Entailment | 0.2148928384990026 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.2828894749305018 |
| Lexical Semantics - Factivity | 0.1781741612749496 |
| Lexical Semantics - Symmetry/Collectivity | 0.22821773229381923 |
| Lexical Semantics - Redundancy | 0.30692490702242564 |
| Lexical Semantics - Named Entities | 0.16692446522239718 |
| Lexical Semantics - Quantifiers | 0.47387910220727386 |
| Predicate-Argument Structure Core Args | 0.3304857994026801 |
| Predicate-Argument Structure Prepositional Phrases | 0.26354595321588503 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.23570226039551584 |
| Predicate-Argument Structure Anaphora/Coreference | -0.002651969513727742 |
| Predicate-Argument Structure Active/Passive | 0.1382147381437882 |
| Predicate-Argument Structure Nominalization | 0.24552983583446694 |
| Predicate-Argument Structure Genitives/Partitives | 0.10482848367219183 |
| Predicate-Argument Structure Datives | 0.3563483225498992 |
| Predicate-Argument Structure Relative Clauses | 0.01642880193633814 |
| Predicate-Argument Structure Coordination Scopes | 0.17910620335162064 |
| Predicate-Argument Structure Intersectivity | 0.4129955231527934 |
| Predicate-Argument Structure Restrictivity | -0.1895424836601307 |
| Logic Negation | 0.09642916726065977 |
| Logic Double Negation | 0.2727272727272727 |
| Logic Interval/Numbers | -0.019157088122605363 |
| Logic Conjuction | 0.10540925533894599 |
| Logic Disjunction | 0.1286978904175574 |
| Logic Conditionals | 0.14285714285714285 |
| Logic Universal | 0.26856632724128343 |
| Logic Existential | 0.060522753266880246 |
| Logic Temporal | 0.12434118282549846 |
| Logic Upward Monotone | 0.3398363161942986 |
| Logic Downward Monotone | 0.10984700727621793 |
| Logic Non-Monotonic | -0.2163856337073152 |
| Knowledge Common Sense | 0.2514207350602513 |
| Knowledge World Knowledge | 0.1854598701600781 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |