Submission Baseline TF-IDF1.1

Nov. 19, 2020, 8:37 a.m.

Team: AGI NLP

Model url: https://github.com/RussianNLP/RussianSuperGLUE/blob/master/TFIDF%20baseline.ipynb


Total score: 0.434

Dataset Score Metric
LiDiRus 0.06 Matthew`s Corr
RCB 0.301 / 0.441 F1/Acc
PARus 0.486 Accuracy
MuSeRC 0.587 / 0.242 F1a/Em
TERRa 0.471 Accuracy
RUSSE 0.57 Accuracy
RWSD 0.662 Accuracy
DaNetQA 0.621 Accuracy
RuCoS 0.26 / 0.252 F1/EM
Model description:

Baseline for SuperGLUE. tf-idf vectorizer plus linear regression The vectorizer you can download here: https://russiansuperglue.com/tasks/tf_idf


Parameter description:

Baseline for SuperGLUE. All information and scripts are available in our repo.

Diagnostic (Matthew`s Correlation): 0.06

Category Score
LOGIC 0.05030841999523283
KNOWLEDGE -0.02216156828576551
PREDICATE-ARGUMENT STRUCTURE 0.15636378666500897
LEXICAL SEMANTICS 0.02855372552262154
Lexical Semantics - Lexical Entailment -0.12104550653376048
Lexical Semantics - Morphological Negation 0.4147575310031266
Lexical Semantics - Factivity 0.017946063401938852
Lexical Semantics - Symmetry/Collectivity 0.5185449728701348
Lexical Semantics - Redundancy 0.16571045299983225
Lexical Semantics - Named Entities 0.14907119849998599
Lexical Semantics - Quantifiers -0.12284057901591701
Predicate-Argument Structure Core Args 0.49888765156985887
Predicate-Argument Structure Prepositional Phrases 0.21272640939132414
Predicate-Argument Structure Ellipsis/Implicits 0.16735967034880075
Predicate-Argument Structure Anaphora/Coreference 0.13725490196078433
Predicate-Argument Structure Active/Passive 0.006980898471528258
Predicate-Argument Structure Nominalization -0.04303314829119352
Predicate-Argument Structure Genitives/Partitives 0.4082482904638631
Predicate-Argument Structure Datives 0.5345224838248488
Predicate-Argument Structure Relative Clauses 0.1044465935734187
Predicate-Argument Structure Coordination Scopes 0.2058790548922549
Predicate-Argument Structure Intersectivity -0.03933290696693271
Predicate-Argument Structure Restrictivity -0.3099066319908699
Logic Negation 0.07397954428741078
Logic Double Negation -0.3892494720807615
Logic Interval/Numbers -0.16884185384856412
Logic Conjuction 0.0778498944161523
Logic Disjunction 0.006434894520877869
Logic Conditionals 0.2519763153394848
Logic Universal 0.4029114820126901
Logic Existential -0.04279604925109129
Logic Temporal -0.02486823656509969
Logic Upward Monotone -0.05353033790313108
Logic Downward Monotone -0.027461751819054483
Logic Non-Monotonic 0.023762422826087874
Knowledge Common Sense 0.041223736932130446
Knowledge World Knowledge -0.07510447188430078