18 августа 2021 г. 11:49
Команда SberDevices
Ссылка на модель https://huggingface.co/sberbank-ai/ruRoberta-large
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,343 | Кор, коэффициент Мэтью |
| RCB | 0,357 / 0,518 | F1/Точность |
| PARus | 0,722 | Точность |
| MuSeRC | 0,861 / 0,63 | F1a/Em |
| TERRa | 0,801 | Точность |
| RUSSE | 0,748 | Точность |
| RWSD | 0,669 | Точность |
| DaNetQA | 0,82 | Точность |
| RuCoS | 0,87 / 0,867 | F1/EM |
Russian RoBERTa large model. 355ml parameters, trained in 250 GB Russian texts.
| Категория | Результат |
|---|---|
| LOGIC | 0,1820012297621496 |
| KNOWLEDGE | 0,2698822742714764 |
| PREDICATE-ARGUMENT STRUCTURE | 0,34132316093258036 |
| LEXICAL SEMANTICS | 0,489878955484335 |
| Lexical Semantics - Lexical Entailment | 0,4238498361791447 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,6366624736381951 |
| Lexical Semantics - Factivity | 0,289805474447035 |
| Lexical Semantics - Symmetry/Collectivity | 0,5477225575051661 |
| Lexical Semantics - Redundancy | 0,7985494095046906 |
| Lexical Semantics - Named Entities | 0,5427204202399745 |
| Lexical Semantics - Quantifiers | 0,35043832202523123 |
| Predicate-Argument Structure Core Args | 0,5111111111111111 |
| Predicate-Argument Structure Prepositional Phrases | 0,3972493441698867 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,24433144650348906 |
| Predicate-Argument Structure Anaphora/Coreference | 0,07955908541183225 |
| Predicate-Argument Structure Active/Passive | 0,44353708548308607 |
| Predicate-Argument Structure Nominalization | 0,6024640760767093 |
| Predicate-Argument Structure Genitives/Partitives | 0,6875 |
| Predicate-Argument Structure Datives | 0,7637626158259734 |
| Predicate-Argument Structure Relative Clauses | 0,29277002188455997 |
| Predicate-Argument Structure Coordination Scopes | 0,2516299559794226 |
| Predicate-Argument Structure Intersectivity | 0,20679318146353196 |
| Predicate-Argument Structure Restrictivity | 0,09335200560186732 |
| Logic Negation | 0,11166573401651324 |
| Logic Double Negation | 0,4271410714851651 |
| Logic Interval/Numbers | -0,12121248708414586 |
| Logic Conjuction | 0,24575816200113812 |
| Logic Disjunction | -0,09647574015265818 |
| Logic Conditionals | 0,02486823656509969 |
| Logic Universal | 0,5606119105813882 |
| Logic Existential | 0,42365927286816174 |
| Logic Temporal | -0,030303030303030304 |
| Logic Upward Monotone | 0,3612343522752406 |
| Logic Downward Monotone | 0,06726727939963124 |
| Logic Non-Monotonic | 0,2163856337073152 |
| Knowledge Common Sense | 0,21361974637528325 |
| Knowledge World Knowledge | 0,3084094312371165 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |