21 мая 2021 г. 7:12
Команда SberDevices
Ссылка на модель https://huggingface.co/sberbank-ai/sbert_large_nlu_ru
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,209 | Кор, коэффициент Мэтью |
| RCB | 0,371 / 0,452 | F1/Точность |
| PARus | 0,498 | Точность |
| MuSeRC | 0,646 / 0,327 | F1a/Em |
| TERRa | 0,637 | Точность |
| RUSSE | 0,654 | Точность |
| RWSD | 0,662 | Точность |
| DaNetQA | 0,675 | Точность |
| RuCoS | 0,36 / 0,351 | F1/EM |
BERT large model (uncased) for Sentence Embeddings in Russian language. The model is described in this article https://habr.com/ru/company/sberdevices/blog/527576/ For better quality, use mean token embeddings.
| Категория | Результат |
|---|---|
| LOGIC | 0,1512486176089863 |
| KNOWLEDGE | 0,22099457996253677 |
| PREDICATE-ARGUMENT STRUCTURE | 0,21136700857922586 |
| LEXICAL SEMANTICS | 0,18872385767564745 |
| Lexical Semantics - Lexical Entailment | 0,1496798365790657 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,06023386019368342 |
| Lexical Semantics - Factivity | 0,07516460280028288 |
| Lexical Semantics - Symmetry/Collectivity | 0,06846531968814576 |
| Lexical Semantics - Redundancy | 0,20228869496966945 |
| Lexical Semantics - Named Entities | 0,24253562503633297 |
| Lexical Semantics - Quantifiers | 0,3128913589510993 |
| Predicate-Argument Structure Core Args | 0,18788818120175085 |
| Predicate-Argument Structure Prepositional Phrases | 0,30626557529894255 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,24859984160559254 |
| Predicate-Argument Structure Anaphora/Coreference | 0,01692566771665739 |
| Predicate-Argument Structure Active/Passive | 0,21144800897557345 |
| Predicate-Argument Structure Nominalization | 0,34752402342845795 |
| Predicate-Argument Structure Genitives/Partitives | 0,25 |
| Predicate-Argument Structure Datives | 0,3563483225498992 |
| Predicate-Argument Structure Relative Clauses | 0,034815531191139566 |
| Predicate-Argument Structure Coordination Scopes | 0,43463356032809364 |
| Predicate-Argument Structure Intersectivity | 0,4997725278535636 |
| Predicate-Argument Structure Restrictivity | -0,4218307438660538 |
| Logic Negation | 0,08797475626313346 |
| Logic Double Negation | 0,21320071635561044 |
| Logic Interval/Numbers | 0,16781215551988446 |
| Logic Conjuction | 0,25819888974716115 |
| Logic Disjunction | 0,030479873164977293 |
| Logic Conditionals | 0,10776235844876533 |
| Logic Universal | 0,2548235957188128 |
| Logic Existential | -0,014678923792502562 |
| Logic Temporal | -0,01698823971458752 |
| Logic Upward Monotone | 0,4664786588701423 |
| Logic Downward Monotone | -0,16434353297215165 |
| Logic Non-Monotonic | 0,015456116693379658 |
| Knowledge Common Sense | 0,2646763180494774 |
| Knowledge World Knowledge | 0,1872134785058742 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |