23 июня 2021 г. 11:22
Команда SberDevices
Ссылка на модель https://huggingface.co/sberbank-ai/ruBert-base
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,224 | Кор, коэффициент Мэтью |
| RCB | 0,333 / 0,509 | F1/Точность |
| PARus | 0,476 | Точность |
| MuSeRC | 0,742 / 0,399 | F1a/Em |
| TERRa | 0,703 | Точность |
| RUSSE | 0,706 | Точность |
| RWSD | 0,669 | Точность |
| DaNetQA | 0,712 | Точность |
| RuCoS | 0,74 / 0,716 | F1/EM |
Russian BERT-base model by sberdevice team. 178 million parameters
-
| Категория | Результат |
|---|---|
| LOGIC | 0,18476478354942552 |
| KNOWLEDGE | 0,3023234579421925 |
| PREDICATE-ARGUMENT STRUCTURE | 0,2312485376166611 |
| LEXICAL SEMANTICS | 0,16505967264044785 |
| Lexical Semantics - Lexical Entailment | -0,06836290976917324 |
|---|---|
| Lexical Semantics - Morphological Negation | -0,18712029714127992 |
| Lexical Semantics - Factivity | 0,3450327796711771 |
| Lexical Semantics - Symmetry/Collectivity | -0,05 |
| Lexical Semantics - Redundancy | 0,6756639246921762 |
| Lexical Semantics - Named Entities | 0,1690308509457033 |
| Lexical Semantics - Quantifiers | 0,307735338091128 |
| Predicate-Argument Structure Core Args | 0,23195672348226393 |
| Predicate-Argument Structure Prepositional Phrases | 0,2767681202560477 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,39148014634313566 |
| Predicate-Argument Structure Anaphora/Coreference | -0,004901960784313725 |
| Predicate-Argument Structure Active/Passive | 0,2135744251723958 |
| Predicate-Argument Structure Nominalization | -0,20672455764868078 |
| Predicate-Argument Structure Genitives/Partitives | 0,0625 |
| Predicate-Argument Structure Datives | 0,629940788348712 |
| Predicate-Argument Structure Relative Clauses | 0,29277002188455997 |
| Predicate-Argument Structure Coordination Scopes | 0,42365927286816174 |
| Predicate-Argument Structure Intersectivity | 0,19113767193576686 |
| Predicate-Argument Structure Restrictivity | 0,3967460238079361 |
| Logic Negation | 0,242488506275402 |
| Logic Double Negation | -0,1556997888323046 |
| Logic Interval/Numbers | 0,026393586082163197 |
| Logic Conjuction | 0,3892494720807615 |
| Logic Disjunction | 0,2849696992113355 |
| Logic Conditionals | 0,3578300267477955 |
| Logic Universal | 0,4332001127219817 |
| Logic Existential | 0,31868131868131866 |
| Logic Temporal | -0,030303030303030304 |
| Logic Upward Monotone | 0,36893922595746137 |
| Logic Downward Monotone | -0,1721600753908879 |
| Logic Non-Monotonic | 0,05572782125753529 |
| Knowledge Common Sense | 0,2757905068148632 |
| Knowledge World Knowledge | 0,3031586813156664 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |