25 апреля 2024 г. 6:33
Команда Maxim Bolgov
Ссылка на модель https://huggingface.co/Defetya/qwen-14B-saiga
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,334 | Кор, коэффициент Мэтью |
| RCB | 0,442 / 0,482 | F1/Точность |
| PARus | 0,61 | Точность |
| MuSeRC | 0,725 / 0,254 | F1a/Em |
| TERRa | 0,717 | Точность |
| RUSSE | 0,464 | Точность |
| RWSD | 0,695 | Точность |
| DaNetQA | 0,791 | Точность |
| RuCoS | 0,43 / 0,42 | F1/EM |
Qwen 14B by Alibaba, full parameter fine-tuned on Saiga dataset for 1 epoch.
| Категория | Результат |
|---|---|
| LOGIC | 0,26494440817298587 |
| KNOWLEDGE | 0,25632452508630893 |
| PREDICATE-ARGUMENT STRUCTURE | 0,32435006166584013 |
| LEXICAL SEMANTICS | 0,4632212533846751 |
| Lexical Semantics - Lexical Entailment | 0,3781169292734477 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,5476190476190477 |
| Lexical Semantics - Factivity | 0,408248290463863 |
| Lexical Semantics - Symmetry/Collectivity | 0,2921186973360886 |
| Lexical Semantics - Redundancy | 0,6922186552431729 |
| Lexical Semantics - Named Entities | 0,40482045237636816 |
| Lexical Semantics - Quantifiers | 0,4816727030991569 |
| Predicate-Argument Structure Core Args | 0,4778552908698884 |
| Predicate-Argument Structure Prepositional Phrases | 0,26323550158162423 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,225101084147303 |
| Predicate-Argument Structure Anaphora/Coreference | 0,169859653214886 |
| Predicate-Argument Structure Active/Passive | 0,45241392835886407 |
| Predicate-Argument Structure Nominalization | 0,5477225575051662 |
| Predicate-Argument Structure Genitives/Partitives | 0,6666666666666666 |
| Predicate-Argument Structure Datives | 0,641688947919748 |
| Predicate-Argument Structure Relative Clauses | 0,29277002188455997 |
| Predicate-Argument Structure Coordination Scopes | 0,1188643277625491 |
| Predicate-Argument Structure Intersectivity | 0,32489314482696546 |
| Predicate-Argument Structure Restrictivity | 0,0 |
| Logic Negation | 0,34363701612718517 |
| Logic Double Negation | 0,3481553119113957 |
| Logic Interval/Numbers | 0,010300524052492094 |
| Logic Conjuction | 0,20672455764868075 |
| Logic Disjunction | 0,36035409316916067 |
| Logic Conditionals | -0,050964719143762556 |
| Logic Universal | 0,3223291856101521 |
| Logic Existential | 0,48038446141526137 |
| Logic Temporal | 0,3829708431025352 |
| Logic Upward Monotone | 0,3244428422615251 |
| Logic Downward Monotone | 0,07098646750254821 |
| Logic Non-Monotonic | 0,2163856337073152 |
| Knowledge Common Sense | 0,2245231365137664 |
| Knowledge World Knowledge | 0,28078011281367116 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |