19 ноября 2020 г. 8:37
Команда AGI NLP
Ссылка на модель https://github.com/RussianNLP/RussianSuperGLUE/blob/master/TFIDF%20baseline.ipynb
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,06 | Кор, коэффициент Мэтью |
| RCB | 0,301 / 0,441 | F1/Точность |
| PARus | 0,486 | Точность |
| MuSeRC | 0,587 / 0,242 | F1a/Em |
| TERRa | 0,471 | Точность |
| RUSSE | 0,57 | Точность |
| RWSD | 0,662 | Точность |
| DaNetQA | 0,621 | Точность |
| RuCoS | 0,26 / 0,252 | F1/EM |
Baseline for SuperGLUE. tf-idf vectorizer plus linear regression The vectorizer you can download here: https://russiansuperglue.com/tasks/tf_idf
Baseline for SuperGLUE. All information and scripts are available in our repo.
| Категория | Результат |
|---|---|
| LOGIC | 0,05030841999523283 |
| KNOWLEDGE | -0,02216156828576551 |
| PREDICATE-ARGUMENT STRUCTURE | 0,15636378666500897 |
| LEXICAL SEMANTICS | 0,02855372552262154 |
| Lexical Semantics - Lexical Entailment | -0,12104550653376048 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,4147575310031266 |
| Lexical Semantics - Factivity | 0,017946063401938852 |
| Lexical Semantics - Symmetry/Collectivity | 0,5185449728701348 |
| Lexical Semantics - Redundancy | 0,16571045299983225 |
| Lexical Semantics - Named Entities | 0,14907119849998599 |
| Lexical Semantics - Quantifiers | -0,12284057901591701 |
| Predicate-Argument Structure Core Args | 0,49888765156985887 |
| Predicate-Argument Structure Prepositional Phrases | 0,21272640939132414 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,16735967034880075 |
| Predicate-Argument Structure Anaphora/Coreference | 0,13725490196078433 |
| Predicate-Argument Structure Active/Passive | 0,006980898471528258 |
| Predicate-Argument Structure Nominalization | -0,04303314829119352 |
| Predicate-Argument Structure Genitives/Partitives | 0,4082482904638631 |
| Predicate-Argument Structure Datives | 0,5345224838248488 |
| Predicate-Argument Structure Relative Clauses | 0,1044465935734187 |
| Predicate-Argument Structure Coordination Scopes | 0,2058790548922549 |
| Predicate-Argument Structure Intersectivity | -0,03933290696693271 |
| Predicate-Argument Structure Restrictivity | -0,3099066319908699 |
| Logic Negation | 0,07397954428741078 |
| Logic Double Negation | -0,3892494720807615 |
| Logic Interval/Numbers | -0,16884185384856412 |
| Logic Conjuction | 0,0778498944161523 |
| Logic Disjunction | 0,006434894520877869 |
| Logic Conditionals | 0,2519763153394848 |
| Logic Universal | 0,4029114820126901 |
| Logic Existential | -0,04279604925109129 |
| Logic Temporal | -0,02486823656509969 |
| Logic Upward Monotone | -0,05353033790313108 |
| Logic Downward Monotone | -0,027461751819054483 |
| Logic Non-Monotonic | 0,023762422826087874 |
| Knowledge Common Sense | 0,041223736932130446 |
| Knowledge World Knowledge | -0,07510447188430078 |
| Датасет | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |