21 апреля 2024 г. 1:00
Команда Maxim Bolgov
Ссылка на модель https://huggingface.co/Defetya/qwen-4B-saiga
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,274 | Кор, коэффициент Мэтью |
| RCB | 0,361 / 0,493 | F1/Точность |
| PARus | 0,554 | Точность |
| MuSeRC | 0,656 / 0,112 | F1a/Em |
| TERRa | 0,655 | Точность |
| RUSSE | 0,57 | Точность |
| RWSD | 0,623 | Точность |
| DaNetQA | 0,661 | Точность |
| RuCoS | 0,4 / 0,395 | F1/EM |
Qwen 1.5 4B by Alibaba, fully fine-tuned on Ilya Gusev's Saiga dataset. Training as well as evaluation was performed on Google v4-32 TPU by TRC, see https://github.com/defdet/rulm/blob/master/self_instruct/src/benchmarks/eval_rsg_tpu.py for evaluation.
| Категория | Результат |
|---|---|
| LOGIC | 0,1982492693072155 |
| KNOWLEDGE | 0,26419115612435906 |
| PREDICATE-ARGUMENT STRUCTURE | 0,24601845359596022 |
| LEXICAL SEMANTICS | 0,3219335477393724 |
| Lexical Semantics - Lexical Entailment | 0,36425098156923136 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,4147575310031266 |
| Lexical Semantics - Factivity | 0,34641016151377546 |
| Lexical Semantics - Symmetry/Collectivity | -0,12171612389003691 |
| Lexical Semantics - Redundancy | -0,08695652173913043 |
| Lexical Semantics - Named Entities | 0,2891574659831201 |
| Lexical Semantics - Quantifiers | 0,19045970117256472 |
| Predicate-Argument Structure Core Args | 0,2297707831614211 |
| Predicate-Argument Structure Prepositional Phrases | 0,34665482822302573 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,10622957319984967 |
| Predicate-Argument Structure Anaphora/Coreference | 0,1667000100033345 |
| Predicate-Argument Structure Active/Passive | 0,27640672769878033 |
| Predicate-Argument Structure Nominalization | 0,35355339059327373 |
| Predicate-Argument Structure Genitives/Partitives | 0,4900980294098034 |
| Predicate-Argument Structure Datives | 0,5091750772173156 |
| Predicate-Argument Structure Relative Clauses | 0,34097520463683817 |
| Predicate-Argument Structure Coordination Scopes | 0,16834512458535864 |
| Predicate-Argument Structure Intersectivity | 0,302433025214821 |
| Predicate-Argument Structure Restrictivity | -0,009732619374563063 |
| Logic Negation | 0,3208739674079263 |
| Logic Double Negation | 0,3481553119113957 |
| Logic Interval/Numbers | -0,16884185384856412 |
| Logic Conjuction | 0,12909944487358058 |
| Logic Disjunction | 0,41702882811414954 |
| Logic Conditionals | 0,21821789023599236 |
| Logic Universal | 0,15228622596829317 |
| Logic Existential | 0,5384615384615384 |
| Logic Temporal | -0,09449929894737882 |
| Logic Upward Monotone | 0,41130637283031135 |
| Logic Downward Monotone | -0,1504142093990467 |
| Logic Non-Monotonic | 0,017197979012252743 |
| Knowledge Common Sense | 0,19707513098159088 |
| Knowledge World Knowledge | 0,33004157719410765 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |