3 февраля 2023 г. 14:24
Команда SberDevices
Ссылка на модель https://huggingface.co/sberbank-ai/ruElectra-small
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,106 | Кор, коэффициент Мэтью |
| RCB | 0,346 / 0,461 | F1/Точность |
| PARus | 0,564 | Точность |
| MuSeRC | 0,628 / 0,21 | F1a/Em |
| TERRa | 0,54 | Точность |
| RUSSE | 0,592 | Точность |
| RWSD | 0,669 | Точность |
| DaNetQA | 0,658 | Точность |
| RuCoS | 0,6 / 0,596 | F1/EM |
ruElectra-small model of critic encoder class, pretraining with google ELECTRA code. It has 12 layers and hidden size 256. It was trained on a Russian language corpus (100GB). The dataset is the same as for sbert_large_mt_nlu_ru models. Wordpiece tokenizer. This model we use as reward-critic model for RLHF and black-box attack. Source data for training: Taiga, Lenta, OpenSubtitles, Wiki, etc. - all with ru lang. domain. Scoring pipeline: https://github.com/ai-forever/rsg-baselines
| Категория | Результат |
|---|---|
| LOGIC | 0,12277394413781632 |
| KNOWLEDGE | 0,09125545109358636 |
| PREDICATE-ARGUMENT STRUCTURE | 0,13081795368454147 |
| LEXICAL SEMANTICS | 0,10240704512860838 |
| Lexical Semantics - Lexical Entailment | -0,018827751470952853 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,31180478223116176 |
| Lexical Semantics - Factivity | 0,14383899044561524 |
| Lexical Semantics - Symmetry/Collectivity | -0,17541160386140586 |
| Lexical Semantics - Redundancy | 0,0 |
| Lexical Semantics - Named Entities | -0,1690308509457033 |
| Lexical Semantics - Quantifiers | 0,23939494881986928 |
| Predicate-Argument Structure Core Args | 0,016412198797244364 |
| Predicate-Argument Structure Prepositional Phrases | 0,29511937286173096 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,0 |
| Predicate-Argument Structure Anaphora/Coreference | 0,13302660798266094 |
| Predicate-Argument Structure Active/Passive | 0,21151845136150904 |
| Predicate-Argument Structure Nominalization | -0,20672455764868078 |
| Predicate-Argument Structure Genitives/Partitives | 0,0 |
| Predicate-Argument Structure Datives | 0,0 |
| Predicate-Argument Structure Relative Clauses | 0,13912166872805048 |
| Predicate-Argument Structure Coordination Scopes | 0,16834512458535864 |
| Predicate-Argument Structure Intersectivity | 0,1250514297491417 |
| Predicate-Argument Structure Restrictivity | 0,0 |
| Logic Negation | 0,16815001717046676 |
| Logic Double Negation | 0,5310850045437943 |
| Logic Interval/Numbers | 0,09158437663089954 |
| Logic Conjuction | 0,25819888974716115 |
| Logic Disjunction | -0,004191297423079544 |
| Logic Conditionals | 0,03253000243161777 |
| Logic Universal | 0,025482359571881278 |
| Logic Existential | -0,10482848367219183 |
| Logic Temporal | 0,007053982594841415 |
| Logic Upward Monotone | 0,22213082915965962 |
| Logic Downward Monotone | -0,035955873264830976 |
| Logic Non-Monotonic | 0,0 |
| Knowledge Common Sense | 0,08627708544516655 |
| Knowledge World Knowledge | 0,0673108631781093 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |