July 26, 2023, 7:01 p.m.
Team: Saiga team
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.398 | Matthew`s Corr |
| RCB | 0.489 / 0.543 | F1/Acc |
| PARus | 0.784 | Accuracy |
| MuSeRC | 0.919 / 0.761 | F1a/Em |
| TERRa | 0.793 | Accuracy |
| RUSSE | 0.74 | Accuracy |
| RWSD | 0.714 | Accuracy |
| DaNetQA | 0.907 | Accuracy |
| RuCoS | 0.78 / 0.76 | F1/EM |
LLaMA-2-13B tuned with LoRA on all tasks simultaneously, with one more RCB/TERRa/RUSSE/RuCoS adapter. Configuration files: - https://github.com/IlyaGusev/rulm/blob/master/self_instruct/configs/llama2_13b_rsg.json - https://github.com/IlyaGusev/rulm/blob/master/self_instruct/configs/llama2_13b_rsg_rucos.json - https://github.com/IlyaGusev/rulm/blob/master/self_instruct/configs/llama2_13b_rsg_rcb.json - https://github.com/IlyaGusev/rulm/blob/master/self_instruct/configs/llama2_13b_rsg_russe.json - https://github.com/IlyaGusev/rulm/blob/master/self_instruct/configs/llama2_13b_rsg_terra.json Full code: https://github.com/IlyaGusev/rulm/tree/master/self_instruct
| Category | Score |
|---|---|
| LOGIC | 0.3148500034107766 |
| KNOWLEDGE | 0.3039368375224113 |
| PREDICATE-ARGUMENT STRUCTURE | 0.4111375055984087 |
| LEXICAL SEMANTICS | 0.439950224154387 |
| Lexical Semantics - Lexical Entailment | 0.44665868111472085 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.0 |
| Lexical Semantics - Factivity | 0.49660854465066007 |
| Lexical Semantics - Symmetry/Collectivity | 0.3302891295379082 |
| Lexical Semantics - Redundancy | 0.01180480411624714 |
| Lexical Semantics - Named Entities | 0.5427204202399745 |
| Lexical Semantics - Quantifiers | 0.5373581667678433 |
| Predicate-Argument Structure Core Args | 0.37777777777777777 |
| Predicate-Argument Structure Prepositional Phrases | 0.4710370493665035 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.7638888888888888 |
| Predicate-Argument Structure Anaphora/Coreference | 0.3818836099767948 |
| Predicate-Argument Structure Active/Passive | 0.4593525712227975 |
| Predicate-Argument Structure Nominalization | 0.2581988897471611 |
| Predicate-Argument Structure Genitives/Partitives | 0.45226701686664544 |
| Predicate-Argument Structure Datives | 0.42857142857142855 |
| Predicate-Argument Structure Relative Clauses | 0.4879500364742666 |
| Predicate-Argument Structure Coordination Scopes | 0.12087912087912088 |
| Predicate-Argument Structure Intersectivity | 0.5520932474951681 |
| Predicate-Argument Structure Restrictivity | 0.2958081738859997 |
| Logic Negation | 0.21444853852539 |
| Logic Double Negation | 0.1005037815259212 |
| Logic Interval/Numbers | 0.0664473678268354 |
| Logic Conjuction | 0.1868706368604627 |
| Logic Disjunction | 0.42472597855881744 |
| Logic Conditionals | 0.3142857142857143 |
| Logic Universal | 0.5606119105813882 |
| Logic Existential | 0.30261376633440124 |
| Logic Temporal | 0.49517597397212765 |
| Logic Upward Monotone | 0.39847500714897677 |
| Logic Downward Monotone | 0.25065177828520524 |
| Logic Non-Monotonic | 0.38769905726763276 |
| Knowledge Common Sense | 0.2835403618148843 |
| Knowledge World Knowledge | 0.30976191939355957 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |