26 июля 2023 г. 19:01
Команда Saiga team
Ссылка на модель https://huggingface.co/meta-llama/Llama-2-13b-hf
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,398 | Кор, коэффициент Мэтью |
| RCB | 0,489 / 0,543 | F1/Точность |
| PARus | 0,784 | Точность |
| MuSeRC | 0,919 / 0,761 | F1a/Em |
| TERRa | 0,793 | Точность |
| RUSSE | 0,74 | Точность |
| RWSD | 0,714 | Точность |
| DaNetQA | 0,907 | Точность |
| RuCoS | 0,78 / 0,76 | F1/EM |
LLaMA-2-13B tuned with LoRA on all tasks simultaneously, with one more RCB/TERRa/RUSSE/RuCoS adapter. Configuration files: - https://github.com/IlyaGusev/rulm/blob/master/self_instruct/configs/llama2_13b_rsg.json - https://github.com/IlyaGusev/rulm/blob/master/self_instruct/configs/llama2_13b_rsg_rucos.json - https://github.com/IlyaGusev/rulm/blob/master/self_instruct/configs/llama2_13b_rsg_rcb.json - https://github.com/IlyaGusev/rulm/blob/master/self_instruct/configs/llama2_13b_rsg_russe.json - https://github.com/IlyaGusev/rulm/blob/master/self_instruct/configs/llama2_13b_rsg_terra.json Full code: https://github.com/IlyaGusev/rulm/tree/master/self_instruct
| Категория | Результат |
|---|---|
| LOGIC | 0,3148500034107766 |
| KNOWLEDGE | 0,3039368375224113 |
| PREDICATE-ARGUMENT STRUCTURE | 0,4111375055984087 |
| LEXICAL SEMANTICS | 0,439950224154387 |
| Lexical Semantics - Lexical Entailment | 0,44665868111472085 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,0 |
| Lexical Semantics - Factivity | 0,49660854465066007 |
| Lexical Semantics - Symmetry/Collectivity | 0,3302891295379082 |
| Lexical Semantics - Redundancy | 0,01180480411624714 |
| Lexical Semantics - Named Entities | 0,5427204202399745 |
| Lexical Semantics - Quantifiers | 0,5373581667678433 |
| Predicate-Argument Structure Core Args | 0,37777777777777777 |
| Predicate-Argument Structure Prepositional Phrases | 0,4710370493665035 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,7638888888888888 |
| Predicate-Argument Structure Anaphora/Coreference | 0,3818836099767948 |
| Predicate-Argument Structure Active/Passive | 0,4593525712227975 |
| Predicate-Argument Structure Nominalization | 0,2581988897471611 |
| Predicate-Argument Structure Genitives/Partitives | 0,45226701686664544 |
| Predicate-Argument Structure Datives | 0,42857142857142855 |
| Predicate-Argument Structure Relative Clauses | 0,4879500364742666 |
| Predicate-Argument Structure Coordination Scopes | 0,12087912087912088 |
| Predicate-Argument Structure Intersectivity | 0,5520932474951681 |
| Predicate-Argument Structure Restrictivity | 0,2958081738859997 |
| Logic Negation | 0,21444853852539 |
| Logic Double Negation | 0,1005037815259212 |
| Logic Interval/Numbers | 0,0664473678268354 |
| Logic Conjuction | 0,1868706368604627 |
| Logic Disjunction | 0,42472597855881744 |
| Logic Conditionals | 0,3142857142857143 |
| Logic Universal | 0,5606119105813882 |
| Logic Existential | 0,30261376633440124 |
| Logic Temporal | 0,49517597397212765 |
| Logic Upward Monotone | 0,39847500714897677 |
| Logic Downward Monotone | 0,25065177828520524 |
| Logic Non-Monotonic | 0,38769905726763276 |
| Knowledge Common Sense | 0,2835403618148843 |
| Knowledge World Knowledge | 0,30976191939355957 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |