May 21, 2021, 7:12 a.m.
Team: SberDevices
Model url: https://huggingface.co/sberbank-ai/sbert_large_nlu_ru
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.209 | Matthew`s Corr |
| RCB | 0.371 / 0.452 | F1/Acc |
| PARus | 0.498 | Accuracy |
| MuSeRC | 0.646 / 0.327 | F1a/Em |
| TERRa | 0.637 | Accuracy |
| RUSSE | 0.654 | Accuracy |
| RWSD | 0.662 | Accuracy |
| DaNetQA | 0.675 | Accuracy |
| RuCoS | 0.36 / 0.351 | F1/EM |
BERT large model (uncased) for Sentence Embeddings in Russian language. The model is described in this article https://habr.com/ru/company/sberdevices/blog/527576/ For better quality, use mean token embeddings.
| Category | Score |
|---|---|
| LOGIC | 0.1512486176089863 |
| KNOWLEDGE | 0.22099457996253677 |
| PREDICATE-ARGUMENT STRUCTURE | 0.21136700857922586 |
| LEXICAL SEMANTICS | 0.18872385767564745 |
| Lexical Semantics - Lexical Entailment | 0.1496798365790657 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.06023386019368342 |
| Lexical Semantics - Factivity | 0.07516460280028288 |
| Lexical Semantics - Symmetry/Collectivity | 0.06846531968814576 |
| Lexical Semantics - Redundancy | 0.20228869496966945 |
| Lexical Semantics - Named Entities | 0.24253562503633297 |
| Lexical Semantics - Quantifiers | 0.3128913589510993 |
| Predicate-Argument Structure Core Args | 0.18788818120175085 |
| Predicate-Argument Structure Prepositional Phrases | 0.30626557529894255 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.24859984160559254 |
| Predicate-Argument Structure Anaphora/Coreference | 0.01692566771665739 |
| Predicate-Argument Structure Active/Passive | 0.21144800897557345 |
| Predicate-Argument Structure Nominalization | 0.34752402342845795 |
| Predicate-Argument Structure Genitives/Partitives | 0.25 |
| Predicate-Argument Structure Datives | 0.3563483225498992 |
| Predicate-Argument Structure Relative Clauses | 0.034815531191139566 |
| Predicate-Argument Structure Coordination Scopes | 0.43463356032809364 |
| Predicate-Argument Structure Intersectivity | 0.4997725278535636 |
| Predicate-Argument Structure Restrictivity | -0.4218307438660538 |
| Logic Negation | 0.08797475626313346 |
| Logic Double Negation | 0.21320071635561044 |
| Logic Interval/Numbers | 0.16781215551988446 |
| Logic Conjuction | 0.25819888974716115 |
| Logic Disjunction | 0.030479873164977293 |
| Logic Conditionals | 0.10776235844876533 |
| Logic Universal | 0.2548235957188128 |
| Logic Existential | -0.014678923792502562 |
| Logic Temporal | -0.01698823971458752 |
| Logic Upward Monotone | 0.4664786588701423 |
| Logic Downward Monotone | -0.16434353297215165 |
| Logic Non-Monotonic | 0.015456116693379658 |
| Knowledge Common Sense | 0.2646763180494774 |
| Knowledge World Knowledge | 0.1872134785058742 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |