25 апреля 2024 г. 6:45
Команда Maxim Bolgov
Ссылка на модель https://huggingface.co/Defetya/qwen-7B-saiga
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,334 | Кор, коэффициент Мэтью |
| RCB | 0,405 / 0,479 | F1/Точность |
| PARus | 0,576 | Точность |
| MuSeRC | 0,659 / 0,239 | F1a/Em |
| TERRa | 0,707 | Точность |
| RUSSE | 0,547 | Точность |
| RWSD | 0,604 | Точность |
| DaNetQA | 0,728 | Точность |
| RuCoS | 0,29 / 0,284 | F1/EM |
Qwen 7B by Alibaba, full parameter fine-tuned on Ilya Gusev's multi-turn dataset.
| Категория | Результат |
|---|---|
| LOGIC | 0,24129843257807204 |
| KNOWLEDGE | 0,2728973763200917 |
| PREDICATE-ARGUMENT STRUCTURE | 0,33121582113845127 |
| LEXICAL SEMANTICS | 0,43634545018304366 |
| Lexical Semantics - Lexical Entailment | 0,36029344305091476 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,3343669275452117 |
| Lexical Semantics - Factivity | 0,3481005662059056 |
| Lexical Semantics - Symmetry/Collectivity | 0,5887840577551898 |
| Lexical Semantics - Redundancy | 0,6756639246921762 |
| Lexical Semantics - Named Entities | 0,3944053188733077 |
| Lexical Semantics - Quantifiers | 0,3878641469525484 |
| Predicate-Argument Structure Core Args | 0,427617987059879 |
| Predicate-Argument Structure Prepositional Phrases | 0,7034363067682678 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,2986111111111111 |
| Predicate-Argument Structure Anaphora/Coreference | -0,05808091224920428 |
| Predicate-Argument Structure Active/Passive | 0,5322656777660987 |
| Predicate-Argument Structure Nominalization | 0,4687501237868722 |
| Predicate-Argument Structure Genitives/Partitives | 0,45226701686664544 |
| Predicate-Argument Structure Datives | 0,2182178902359924 |
| Predicate-Argument Structure Relative Clauses | 0,11500161355436699 |
| Predicate-Argument Structure Coordination Scopes | 0,42857142857142855 |
| Predicate-Argument Structure Intersectivity | 0,2826510721247563 |
| Predicate-Argument Structure Restrictivity | 0,2556549962824568 |
| Logic Negation | 0,4104841828714906 |
| Logic Double Negation | 0,18090680674665818 |
| Logic Interval/Numbers | 0,054625224104784945 |
| Logic Conjuction | 0,2854496128592251 |
| Logic Disjunction | 0,31212739461154115 |
| Logic Conditionals | 0,2698412698412698 |
| Logic Universal | 0,6446583712203042 |
| Logic Existential | 0,04279604925109129 |
| Logic Temporal | -0,01698823971458752 |
| Logic Upward Monotone | 0,3354002900991854 |
| Logic Downward Monotone | -0,008988968316207744 |
| Logic Non-Monotonic | 0,35807332209046444 |
| Knowledge Common Sense | 0,20919238525745862 |
| Knowledge World Knowledge | 0,3235156344425476 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |