Submission Baseline TF-IDF v1.0

June 11, 2020, 1:14 p.m.

Team: AGI NLP

Model url: https://github.com/RussianNLP/RussianSuperGLUE/blob/master/TFIDF%20baseline.ipynb

Total score: 0.438

Dataset Score Metric
LiDiRus -0.004 Matthew`s Corr
RCB 0.395 / 0.288 F1/Acc
PARus 0.522 Accuracy
MuSeRC 0.477 / 0.03 F1a/Em
TERRa 0.496 Accuracy
RUSSE 0.632 Accuracy
RWSD 0.66 Accuracy
DaNetQA 0.763 Accuracy
RuCoS 0.286 / 0.28 F1/EM
Model description:

Baseline for SuperGLUE. tf-idf vectorizer plus linear regression The vectorizer you can download here: https://russiansuperglue.com/tasks/tf_idf


Parameter description:

Baseline for SuperGLUE. All information and scripts are available in our repo.

Diagnostic (Matthew`s Correlation): -0.004

Category Score
LOGIC -0.0439948022739605
KNOWLEDGE -0.0152697506968919
PREDICATE-ARGUMENT STRUCTURE 0.00232932189184149
LEXICAL SEMANTICS 0.031999521336184
Lexical Semantics - Lexical Entailment -0.0309205330871067
Lexical Semantics - Morphological Negation 0.0
Lexical Semantics - Factivity 0.206724557648681
Lexical Semantics - Symmetry/Collectivity 0.175411603861406
Lexical Semantics - Redundancy 0.0
Lexical Semantics - Named Entities 0.0
Lexical Semantics - Quantifiers 0.0
Predicate-Argument Structure Core Args 0.192450089729875
Predicate-Argument Structure Prepositional Phrases -0.118627545998008
Predicate-Argument Structure Ellipsis/Implicits 0.0
Predicate-Argument Structure Anaphora/Coreference 0.0330785077515602
Predicate-Argument Structure Active/Passive -0.154672056222436
Predicate-Argument Structure Nominalization 0.0
Predicate-Argument Structure Genitives/Partitives 0.0
Predicate-Argument Structure Datives 0.0
Predicate-Argument Structure Relative Clauses 0.0
Predicate-Argument Structure Coordination Scopes 0.0
Predicate-Argument Structure Intersectivity 0.0376521663149937
Predicate-Argument Structure Restrictivity 0.0
Logic Negation -0.0365331082900794
Logic Double Negation 0.0
Logic Interval/Numbers -0.163098084665869
Logic Conjuction 0.0
Logic Disjunction 0.0
Logic Conditionals -0.203653269990639
Logic Universal 0.0
Logic Existential 0.0
Logic Temporal -0.0105356590206836
Logic Upward Monotone 0.0296174438879546
Logic Downward Monotone 0.0
Logic Non-Monotonic 0.0
Knowledge Common Sense -0.0632320049394676
Knowledge World Knowledge -0.0336069007806704