16 октября 2023 г. 11:41
Команда SberDevices
Ссылка на модель https://huggingface.co/ai-forever/ruElectra-large
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,197 | Кор, коэффициент Мэтью |
| RCB | 0,386 / 0,459 | F1/Точность |
| PARus | 0,644 | Точность |
| MuSeRC | 0,549 / 0,078 | F1a/Em |
| TERRa | 0,583 | Точность |
| RUSSE | 0,632 | Точность |
| RWSD | 0,669 | Точность |
| DaNetQA | 0,627 | Точность |
| RuCoS | 0,61 / 0,607 | F1/EM |
ruElectra-large model of critic encoder class. It has 24 layers and hidden size 1024. It was trained on a Russian language corpus (100GB). Wordpiece tokenizer. Source data for training: Taiga, Lenta, OpenSubtitles, Wiki, etc. - all with ru lang. domain. Scoring pipeline: https://github.com/ai-forever/rsg-baselines
| Категория | Результат |
|---|---|
| LOGIC | 0,10344882418344328 |
| KNOWLEDGE | 0,18446096974096243 |
| PREDICATE-ARGUMENT STRUCTURE | 0,2395746447136374 |
| LEXICAL SEMANTICS | 0,23707730280981554 |
| Lexical Semantics - Lexical Entailment | 0,1448057711943421 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,21957751641341997 |
| Lexical Semantics - Factivity | 0,16012815380508713 |
| Lexical Semantics - Symmetry/Collectivity | 0,09128709291752768 |
| Lexical Semantics - Redundancy | 0,0 |
| Lexical Semantics - Named Entities | 0,24806946917841693 |
| Lexical Semantics - Quantifiers | 0,2140767569329586 |
| Predicate-Argument Structure Core Args | 0,15052428411666496 |
| Predicate-Argument Structure Prepositional Phrases | 0,3224469744632013 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,24514516892273003 |
| Predicate-Argument Structure Anaphora/Coreference | 0,19308582308458774 |
| Predicate-Argument Structure Active/Passive | 0,24433144650348906 |
| Predicate-Argument Structure Nominalization | 0,12171612389003691 |
| Predicate-Argument Structure Genitives/Partitives | 0,2182178902359924 |
| Predicate-Argument Structure Datives | 0,2182178902359924 |
| Predicate-Argument Structure Relative Clauses | 0,2277100170213244 |
| Predicate-Argument Structure Coordination Scopes | 0,3144854510165755 |
| Predicate-Argument Structure Intersectivity | 0,13429295209341927 |
| Predicate-Argument Structure Restrictivity | 0,17712297710801908 |
| Logic Negation | -0,17437145811572893 |
| Logic Double Negation | 0,45226701686664544 |
| Logic Interval/Numbers | -0,040522044923655395 |
| Logic Conjuction | 0,15491933384829668 |
| Logic Disjunction | 0,07460921467460303 |
| Logic Conditionals | 0,0849411985729376 |
| Logic Universal | 0,3959441875175622 |
| Logic Existential | -0,022875450543583874 |
| Logic Temporal | -0,34188172937891387 |
| Logic Upward Monotone | 0,1604088343952626 |
| Logic Downward Monotone | 0,049773755656108594 |
| Logic Non-Monotonic | 0,2163856337073152 |
| Knowledge Common Sense | 0,13232776605838764 |
| Knowledge World Knowledge | 0,20845916940990206 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |