Feb. 3, 2023, 2:24 p.m.
Team: SberDevices
Model url: https://huggingface.co/sberbank-ai/ruElectra-small
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.106 | Matthew`s Corr |
| RCB | 0.346 / 0.461 | F1/Acc |
| PARus | 0.564 | Accuracy |
| MuSeRC | 0.628 / 0.21 | F1a/Em |
| TERRa | 0.54 | Accuracy |
| RUSSE | 0.592 | Accuracy |
| RWSD | 0.669 | Accuracy |
| DaNetQA | 0.658 | Accuracy |
| RuCoS | 0.6 / 0.596 | F1/EM |
ruElectra-small model of critic encoder class, pretraining with google ELECTRA code. It has 12 layers and hidden size 256. It was trained on a Russian language corpus (100GB). The dataset is the same as for sbert_large_mt_nlu_ru models. Wordpiece tokenizer. This model we use as reward-critic model for RLHF and black-box attack. Source data for training: Taiga, Lenta, OpenSubtitles, Wiki, etc. - all with ru lang. domain. Scoring pipeline: https://github.com/ai-forever/rsg-baselines
| Category | Score |
|---|---|
| LOGIC | 0.12277394413781632 |
| KNOWLEDGE | 0.09125545109358636 |
| PREDICATE-ARGUMENT STRUCTURE | 0.13081795368454147 |
| LEXICAL SEMANTICS | 0.10240704512860838 |
| Lexical Semantics - Lexical Entailment | -0.018827751470952853 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.31180478223116176 |
| Lexical Semantics - Factivity | 0.14383899044561524 |
| Lexical Semantics - Symmetry/Collectivity | -0.17541160386140586 |
| Lexical Semantics - Redundancy | 0.0 |
| Lexical Semantics - Named Entities | -0.1690308509457033 |
| Lexical Semantics - Quantifiers | 0.23939494881986928 |
| Predicate-Argument Structure Core Args | 0.016412198797244364 |
| Predicate-Argument Structure Prepositional Phrases | 0.29511937286173096 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.0 |
| Predicate-Argument Structure Anaphora/Coreference | 0.13302660798266094 |
| Predicate-Argument Structure Active/Passive | 0.21151845136150904 |
| Predicate-Argument Structure Nominalization | -0.20672455764868078 |
| Predicate-Argument Structure Genitives/Partitives | 0.0 |
| Predicate-Argument Structure Datives | 0.0 |
| Predicate-Argument Structure Relative Clauses | 0.13912166872805048 |
| Predicate-Argument Structure Coordination Scopes | 0.16834512458535864 |
| Predicate-Argument Structure Intersectivity | 0.1250514297491417 |
| Predicate-Argument Structure Restrictivity | 0.0 |
| Logic Negation | 0.16815001717046676 |
| Logic Double Negation | 0.5310850045437943 |
| Logic Interval/Numbers | 0.09158437663089954 |
| Logic Conjuction | 0.25819888974716115 |
| Logic Disjunction | -0.004191297423079544 |
| Logic Conditionals | 0.03253000243161777 |
| Logic Universal | 0.025482359571881278 |
| Logic Existential | -0.10482848367219183 |
| Logic Temporal | 0.007053982594841415 |
| Logic Upward Monotone | 0.22213082915965962 |
| Logic Downward Monotone | -0.035955873264830976 |
| Logic Non-Monotonic | 0.0 |
| Knowledge Common Sense | 0.08627708544516655 |
| Knowledge World Knowledge | 0.0673108631781093 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |