20 сентября 2021 г. 6:57
Команда SberDevices
Ссылка на модель https://huggingface.co/sberbank-ai/ruT5-large
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,32 | Кор, коэффициент Мэтью |
| RCB | 0,45 / 0,532 | F1/Точность |
| PARus | 0,764 | Точность |
| MuSeRC | 0,855 / 0,608 | F1a/Em |
| TERRa | 0,775 | Точность |
| RUSSE | 0,773 | Точность |
| RWSD | 0,669 | Точность |
| DaNetQA | 0,79 | Точность |
| RuCoS | 0,86 / 0,859 | F1/EM |
Russian T5-large model. 740 million parameters, trained on 300G texts of different domains.
-
| Категория | Результат |
|---|---|
| LOGIC | 0,20694602549177668 |
| KNOWLEDGE | 0,2617642051708644 |
| PREDICATE-ARGUMENT STRUCTURE | 0,3005668454878599 |
| LEXICAL SEMANTICS | 0,41830327755878044 |
| Lexical Semantics - Lexical Entailment | 0,3727642044390084 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,6172133998483676 |
| Lexical Semantics - Factivity | 0,2721655269759087 |
| Lexical Semantics - Symmetry/Collectivity | 0,43852900965351466 |
| Lexical Semantics - Redundancy | 0,5247497678328021 |
| Lexical Semantics - Named Entities | 0,40482045237636816 |
| Lexical Semantics - Quantifiers | 0,37854910518078255 |
| Predicate-Argument Structure Core Args | 0,47544348941583686 |
| Predicate-Argument Structure Prepositional Phrases | 0,4454180314547771 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,1649915822768611 |
| Predicate-Argument Structure Anaphora/Coreference | 0,15646620130993677 |
| Predicate-Argument Structure Active/Passive | 0,23461612570453771 |
| Predicate-Argument Structure Nominalization | 0,3373495424699933 |
| Predicate-Argument Structure Genitives/Partitives | 0,15724272550828775 |
| Predicate-Argument Structure Datives | 0,43463356032809364 |
| Predicate-Argument Structure Relative Clauses | 0,29814239699997197 |
| Predicate-Argument Structure Coordination Scopes | 0,33454829277463405 |
| Predicate-Argument Structure Intersectivity | 0,3107299650387684 |
| Predicate-Argument Structure Restrictivity | 0,09335200560186732 |
| Logic Negation | -0,09103804161885289 |
| Logic Double Negation | 0,3892494720807615 |
| Logic Interval/Numbers | 0,12121248708414586 |
| Logic Conjuction | 0,38666666666666666 |
| Logic Disjunction | 0,20430156738209967 |
| Logic Conditionals | -0,050964719143762556 |
| Logic Universal | 0,4029114820126901 |
| Logic Existential | 0,3851644432598216 |
| Logic Temporal | 0,11396057645963795 |
| Logic Upward Monotone | 0,6701754385964912 |
| Logic Downward Monotone | -0,24715576637149037 |
| Logic Non-Monotonic | 0,15445574836957118 |
| Knowledge Common Sense | 0,31542785259678374 |
| Knowledge World Knowledge | 0,1866948732559256 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |