Nov. 19, 2020, 8:37 a.m.
Team: AGI NLP
Model url: https://github.com/RussianNLP/RussianSuperGLUE/blob/master/TFIDF%20baseline.ipynb
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.06 | Matthew`s Corr |
| RCB | 0.301 / 0.441 | F1/Acc |
| PARus | 0.486 | Accuracy |
| MuSeRC | 0.587 / 0.242 | F1a/Em |
| TERRa | 0.471 | Accuracy |
| RUSSE | 0.57 | Accuracy |
| RWSD | 0.662 | Accuracy |
| DaNetQA | 0.621 | Accuracy |
| RuCoS | 0.26 / 0.252 | F1/EM |
Baseline for SuperGLUE. tf-idf vectorizer plus linear regression The vectorizer you can download here: https://russiansuperglue.com/tasks/tf_idf
Baseline for SuperGLUE. All information and scripts are available in our repo.
| Category | Score |
|---|---|
| LOGIC | 0.05030841999523283 |
| KNOWLEDGE | -0.02216156828576551 |
| PREDICATE-ARGUMENT STRUCTURE | 0.15636378666500897 |
| LEXICAL SEMANTICS | 0.02855372552262154 |
| Lexical Semantics - Lexical Entailment | -0.12104550653376048 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.4147575310031266 |
| Lexical Semantics - Factivity | 0.017946063401938852 |
| Lexical Semantics - Symmetry/Collectivity | 0.5185449728701348 |
| Lexical Semantics - Redundancy | 0.16571045299983225 |
| Lexical Semantics - Named Entities | 0.14907119849998599 |
| Lexical Semantics - Quantifiers | -0.12284057901591701 |
| Predicate-Argument Structure Core Args | 0.49888765156985887 |
| Predicate-Argument Structure Prepositional Phrases | 0.21272640939132414 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.16735967034880075 |
| Predicate-Argument Structure Anaphora/Coreference | 0.13725490196078433 |
| Predicate-Argument Structure Active/Passive | 0.006980898471528258 |
| Predicate-Argument Structure Nominalization | -0.04303314829119352 |
| Predicate-Argument Structure Genitives/Partitives | 0.4082482904638631 |
| Predicate-Argument Structure Datives | 0.5345224838248488 |
| Predicate-Argument Structure Relative Clauses | 0.1044465935734187 |
| Predicate-Argument Structure Coordination Scopes | 0.2058790548922549 |
| Predicate-Argument Structure Intersectivity | -0.03933290696693271 |
| Predicate-Argument Structure Restrictivity | -0.3099066319908699 |
| Logic Negation | 0.07397954428741078 |
| Logic Double Negation | -0.3892494720807615 |
| Logic Interval/Numbers | -0.16884185384856412 |
| Logic Conjuction | 0.0778498944161523 |
| Logic Disjunction | 0.006434894520877869 |
| Logic Conditionals | 0.2519763153394848 |
| Logic Universal | 0.4029114820126901 |
| Logic Existential | -0.04279604925109129 |
| Logic Temporal | -0.02486823656509969 |
| Logic Upward Monotone | -0.05353033790313108 |
| Logic Downward Monotone | -0.027461751819054483 |
| Logic Non-Monotonic | 0.023762422826087874 |
| Knowledge Common Sense | 0.041223736932130446 |
| Knowledge World Knowledge | -0.07510447188430078 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |