Submission Saiga Mistral 7B zero-shot

Oct. 9, 2023, 5:38 p.m.

Team: Saiga team

Model url: https://huggingface.co/IlyaGusev/saiga_mistral_7b_lora


Total score: 0.635

Dataset Score Metric
LiDiRus 0.322 Matthew`s Corr
RCB 0.436 / 0.5 F1/Acc
PARus 0.698 Accuracy
MuSeRC 0.84 / 0.553 F1a/Em
TERRa 0.807 Accuracy
RUSSE 0.587 Accuracy
RWSD 0.727 Accuracy
DaNetQA 0.839 Accuracy
RuCoS 0.58 / 0.571 F1/EM
Model description:

Saiga Mistral 7B zero-shot Mistral 7B (https://huggingface.co/mistralai/Mistral-7B-v0.1) was tuned on 5 Russian datasets: ru_turbo_saiga, ru_sharegpt_cleaned, oasst1_ru_main_branch, gpt_roleplay_realm, ru_instruct_gpt4 See more information on HuggingFace: https://huggingface.co/IlyaGusev/saiga_mistral_7b_lora Zero-shot evaluation script for the inference for Saiga Mistral model: https://github.com/IlyaGusev/rulm/blob/master/self_instruct/src/benchmarks/eval_zs_rsg.py


Parameter description:

Diagnostic (Matthew`s Correlation): 0.322

Category Score
LOGIC 0.24068909266898375
KNOWLEDGE 0.3467006882891795
PREDICATE-ARGUMENT STRUCTURE 0.32835497921408513
LEXICAL SEMANTICS 0.38897213279554893
Lexical Semantics - Lexical Entailment 0.39133162093562956
Lexical Semantics - Morphological Negation 0.4816727030991569
Lexical Semantics - Factivity 0.16140004156875074
Lexical Semantics - Symmetry/Collectivity 0.43852900965351466
Lexical Semantics - Redundancy 0.6922186552431729
Lexical Semantics - Named Entities 0.4472135954999579
Lexical Semantics - Quantifiers 0.404651319111256
Predicate-Argument Structure Core Args 0.40987803063838396
Predicate-Argument Structure Prepositional Phrases 0.3020666145593326
Predicate-Argument Structure Ellipsis/Implicits 0.39148014634313566
Predicate-Argument Structure Anaphora/Coreference 0.33606722016672236
Predicate-Argument Structure Active/Passive 0.3244428422615251
Predicate-Argument Structure Nominalization 0.6255432421712243
Predicate-Argument Structure Genitives/Partitives 0.6666666666666666
Predicate-Argument Structure Datives 0.35043832202523123
Predicate-Argument Structure Relative Clauses 0.13912166872805048
Predicate-Argument Structure Coordination Scopes 0.0
Predicate-Argument Structure Intersectivity 0.32489314482696546
Predicate-Argument Structure Restrictivity 0.2748737083745107
Logic Negation 0.23037198846919968
Logic Double Negation 0.19312181983410703
Logic Interval/Numbers 0.033731512431528755
Logic Conjuction 0.25819888974716115
Logic Disjunction 0.2364331218717302
Logic Conditionals 0.3646984043128985
Logic Universal 0.15228622596829317
Logic Existential 0.24459979523511427
Logic Temporal 0.007053982594841415
Logic Upward Monotone 0.27640672769878033
Logic Downward Monotone 0.16238586255274184
Logic Non-Monotonic 0.18389242812245682
Knowledge Common Sense 0.3170077616914835
Knowledge World Knowledge 0.37712430952994663

Performance:

Dataset Speed RAM
LiDiRus - -
RCB - -
PARus - -
MuSeRC - -
TERRa - -
RUSSE - -
RWSD - -
DaNetQA - -
RuCoS - -