April 25, 2024, 6:45 a.m.
Team: Maxim Bolgov
Model url: https://huggingface.co/Defetya/qwen-7B-saiga
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.334 | Matthew`s Corr |
| RCB | 0.405 / 0.479 | F1/Acc |
| PARus | 0.576 | Accuracy |
| MuSeRC | 0.659 / 0.239 | F1a/Em |
| TERRa | 0.707 | Accuracy |
| RUSSE | 0.547 | Accuracy |
| RWSD | 0.604 | Accuracy |
| DaNetQA | 0.728 | Accuracy |
| RuCoS | 0.29 / 0.284 | F1/EM |
Qwen 7B by Alibaba, full parameter fine-tuned on Ilya Gusev's multi-turn dataset.
| Category | Score |
|---|---|
| LOGIC | 0.24129843257807204 |
| KNOWLEDGE | 0.2728973763200917 |
| PREDICATE-ARGUMENT STRUCTURE | 0.33121582113845127 |
| LEXICAL SEMANTICS | 0.43634545018304366 |
| Lexical Semantics - Lexical Entailment | 0.36029344305091476 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.3343669275452117 |
| Lexical Semantics - Factivity | 0.3481005662059056 |
| Lexical Semantics - Symmetry/Collectivity | 0.5887840577551898 |
| Lexical Semantics - Redundancy | 0.6756639246921762 |
| Lexical Semantics - Named Entities | 0.3944053188733077 |
| Lexical Semantics - Quantifiers | 0.3878641469525484 |
| Predicate-Argument Structure Core Args | 0.427617987059879 |
| Predicate-Argument Structure Prepositional Phrases | 0.7034363067682678 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.2986111111111111 |
| Predicate-Argument Structure Anaphora/Coreference | -0.05808091224920428 |
| Predicate-Argument Structure Active/Passive | 0.5322656777660987 |
| Predicate-Argument Structure Nominalization | 0.4687501237868722 |
| Predicate-Argument Structure Genitives/Partitives | 0.45226701686664544 |
| Predicate-Argument Structure Datives | 0.2182178902359924 |
| Predicate-Argument Structure Relative Clauses | 0.11500161355436699 |
| Predicate-Argument Structure Coordination Scopes | 0.42857142857142855 |
| Predicate-Argument Structure Intersectivity | 0.2826510721247563 |
| Predicate-Argument Structure Restrictivity | 0.2556549962824568 |
| Logic Negation | 0.4104841828714906 |
| Logic Double Negation | 0.18090680674665818 |
| Logic Interval/Numbers | 0.054625224104784945 |
| Logic Conjuction | 0.2854496128592251 |
| Logic Disjunction | 0.31212739461154115 |
| Logic Conditionals | 0.2698412698412698 |
| Logic Universal | 0.6446583712203042 |
| Logic Existential | 0.04279604925109129 |
| Logic Temporal | -0.01698823971458752 |
| Logic Upward Monotone | 0.3354002900991854 |
| Logic Downward Monotone | -0.008988968316207744 |
| Logic Non-Monotonic | 0.35807332209046444 |
| Knowledge Common Sense | 0.20919238525745862 |
| Knowledge World Knowledge | 0.3235156344425476 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |