27 января 2021 г. 12:11
Команда SberDevices
Ссылка на модель https://huggingface.co/sberbank-ai/rugpt3xl
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,096 | Кор, коэффициент Мэтью |
| RCB | 0,302 / 0,418 | F1/Точность |
| PARus | 0,676 | Точность |
| MuSeRC | 0,74 / 0,546 | F1a/Em |
| TERRa | 0,573 | Точность |
| RUSSE | 0,565 | Точность |
| RWSD | 0,649 | Точность |
| DaNetQA | 0,59 | Точность |
| RuCoS | 0,67 / 0,665 | F1/EM |
This submission was made by Few-Shot methods only ruGPT-3 Xl - 1.3 billion parameters All tasks solved by ranging the perplexity of the answer variants Checkpoints and documentation can be found here: https://github.com/sberbank-ai/ru-gpts https://github.com/sberbank-ai/ru-gpts/tree/master/gw
| Категория | Результат |
|---|---|
| LOGIC | -0,007250629987786855 |
| KNOWLEDGE | 0,059773386339694506 |
| PREDICATE-ARGUMENT STRUCTURE | 0,13997903121271693 |
| LEXICAL SEMANTICS | 0,18737835348183993 |
| Lexical Semantics - Lexical Entailment | 0,029950995334911953 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,0 |
| Lexical Semantics - Factivity | 0,06900655593423542 |
| Lexical Semantics - Symmetry/Collectivity | 0,2921186973360886 |
| Lexical Semantics - Redundancy | 0,45652173913043476 |
| Lexical Semantics - Named Entities | 0,24253562503633297 |
| Lexical Semantics - Quantifiers | 0,20823613034492347 |
| Predicate-Argument Structure Core Args | 0,18433668044583404 |
| Predicate-Argument Structure Prepositional Phrases | 0,23123776351036351 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,08554415042912275 |
| Predicate-Argument Structure Anaphora/Coreference | -0,01692566771665739 |
| Predicate-Argument Structure Active/Passive | 0,05353033790313108 |
| Predicate-Argument Structure Nominalization | 0,0 |
| Predicate-Argument Structure Genitives/Partitives | -0,25 |
| Predicate-Argument Structure Datives | 0,3563483225498992 |
| Predicate-Argument Structure Relative Clauses | 0,09759000729485333 |
| Predicate-Argument Structure Coordination Scopes | 0,35186577527449836 |
| Predicate-Argument Structure Intersectivity | 0,12100224017315647 |
| Predicate-Argument Structure Restrictivity | -0,1455213750217998 |
| Logic Negation | 0,0399615133948266 |
| Logic Double Negation | 0,21102672654165874 |
| Logic Interval/Numbers | -0,25225301366160086 |
| Logic Conjuction | -0,1084652289093281 |
| Logic Disjunction | 0,15291057030852148 |
| Logic Conditionals | -0,14285714285714285 |
| Logic Universal | 0,2548235957188128 |
| Logic Existential | -0,15724272550828775 |
| Logic Temporal | -0,13261933174731558 |
| Logic Upward Monotone | 0,10054155780296499 |
| Logic Downward Monotone | 0,03008284187980934 |
| Logic Non-Monotonic | -0,34151450937027694 |
| Knowledge Common Sense | -0,03115501519756839 |
| Knowledge World Knowledge | 0,14184850482497574 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |