18 августа 2021 г. 11:55
Команда SberDevices
Ссылка на модель https://huggingface.co/sberbank-ai/ruBert-large
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,235 | Кор, коэффициент Мэтью |
| RCB | 0,356 / 0,5 | F1/Точность |
| PARus | 0,656 | Точность |
| MuSeRC | 0,778 / 0,436 | F1a/Em |
| TERRa | 0,704 | Точность |
| RUSSE | 0,707 | Точность |
| RWSD | 0,669 | Точность |
| DaNetQA | 0,773 | Точность |
| RuCoS | 0,81 / 0,805 | F1/EM |
Russian Bert-large model. 430ml parameters, trained on 30 GB Russian texts.
-
| Категория | Результат |
|---|---|
| LOGIC | 0,13965594076204224 |
| KNOWLEDGE | 0,25420452490107115 |
| PREDICATE-ARGUMENT STRUCTURE | 0,2110557307641125 |
| LEXICAL SEMANTICS | 0,23178921890059834 |
| Lexical Semantics - Lexical Entailment | 0,1605357781060343 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,060993754559283325 |
| Lexical Semantics - Factivity | 0,21320071635561044 |
| Lexical Semantics - Symmetry/Collectivity | 0,0 |
| Lexical Semantics - Redundancy | 0,45652173913043476 |
| Lexical Semantics - Named Entities | 0,1767766952966369 |
| Lexical Semantics - Quantifiers | 0,1913754029921869 |
| Predicate-Argument Structure Core Args | 0,33895960971961925 |
| Predicate-Argument Structure Prepositional Phrases | 0,44921394657019476 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,14818474149146235 |
| Predicate-Argument Structure Anaphora/Coreference | 0,07001400420140048 |
| Predicate-Argument Structure Active/Passive | 0,3059179896521843 |
| Predicate-Argument Structure Nominalization | 0,0 |
| Predicate-Argument Structure Genitives/Partitives | 0,25 |
| Predicate-Argument Structure Datives | 0,37796447300922725 |
| Predicate-Argument Structure Relative Clauses | 0,13912166872805048 |
| Predicate-Argument Structure Coordination Scopes | 0,20894746626614943 |
| Predicate-Argument Structure Intersectivity | 0,03545133231866123 |
| Predicate-Argument Structure Restrictivity | -0,1455213750217998 |
| Logic Negation | 0,08219949365267865 |
| Logic Double Negation | 0,0 |
| Logic Interval/Numbers | 0,2876779808912305 |
| Logic Conjuction | 0,29370130064810657 |
| Logic Disjunction | 0,2780367699071248 |
| Logic Conditionals | 0,14285714285714285 |
| Logic Universal | 0,5324675324675324 |
| Logic Existential | 0,10482848367219183 |
| Logic Temporal | -0,12434118282549846 |
| Logic Upward Monotone | 0,31353483628976775 |
| Logic Downward Monotone | -0,09513029883089882 |
| Logic Non-Monotonic | 0,0 |
| Knowledge Common Sense | 0,3022898014232672 |
| Knowledge World Knowledge | 0,17129218721431935 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |