Submission heuristic majority

Feb. 8, 2021, 4:58 p.m.

Team: hse_ling


Total score: 0.468

Dataset Score Metric
LiDiRus 0.147 Matthew`s Corr
RCB 0.4 / 0.438 F1/Acc
PARus 0.478 Accuracy
MuSeRC 0.671 / 0.237 F1a/Em
TERRa 0.549 Accuracy
RUSSE 0.595 Accuracy
RWSD 0.669 Accuracy
DaNetQA 0.642 Accuracy
RuCoS 0.26 / 0.257 F1/EM
Model description:

Simple rule-based heuristics like: full match of the instance: (RCB) premise: "Из материалов дела следует, что начальник одного из отделов пытался выбить субсидию заинтересованной организации за «откат»." hypothesis: "Начальник одного из отделов пытался выбить субсидию заинтересованной организации за «откат»." idx (training data): 12 instance's length: (TERRa) if premise has less than 29 words in it's likely that the label is "not_entailment" same form: (RUSSE) if word forms are the same it is likely that the label is "False". For instances where an heurstic does not work, the label was predicted with the help of majority class (based on training and validation datasets combined)


Parameter description:

Diagnostic (Matthew`s Correlation): 0.147

Category Score
LOGIC 0.20234318712510463
KNOWLEDGE 0.11732351833491794
PREDICATE-ARGUMENT STRUCTURE 0.13699734914800707
LEXICAL SEMANTICS 0.17542964532360103
Lexical Semantics - Lexical Entailment 0.1431860686204434
Lexical Semantics - Morphological Negation 0.14085904245475275
Lexical Semantics - Factivity 0.31622776601683794
Lexical Semantics - Symmetry/Collectivity -0.41079191812887456
Lexical Semantics - Redundancy 0.2211629342323457
Lexical Semantics - Named Entities 0.0
Lexical Semantics - Quantifiers 0.3295017884191656
Predicate-Argument Structure Core Args -0.11013775835078694
Predicate-Argument Structure Prepositional Phrases 0.37605071654517747
Predicate-Argument Structure Ellipsis/Implicits 0.16735967034880075
Predicate-Argument Structure Anaphora/Coreference 0.05945643676931176
Predicate-Argument Structure Active/Passive -0.22213082915965962
Predicate-Argument Structure Nominalization 0.1414213562373095
Predicate-Argument Structure Genitives/Partitives 0.16666666666666666
Predicate-Argument Structure Datives 0.3273268353539886
Predicate-Argument Structure Relative Clauses -0.09759000729485333
Predicate-Argument Structure Coordination Scopes 0.3126409456585232
Predicate-Argument Structure Intersectivity 0.21119243915593267
Predicate-Argument Structure Restrictivity 0.2627807231132027
Logic Negation 0.16891891891891891
Logic Double Negation 0.3302891295379082
Logic Interval/Numbers -0.13130643285972254
Logic Conjuction 0.08606629658238704
Logic Disjunction 0.29186501192363773
Logic Conditionals 0.32277655457716287
Logic Universal 0.08058229640253803
Logic Existential 0.10482848367219183
Logic Temporal 0.07984636960487998
Logic Upward Monotone 0.21022325722438134
Logic Downward Monotone 0.035955873264830976
Logic Non-Monotonic 0.3152441624956403
Knowledge Common Sense 0.15188811433521146
Knowledge World Knowledge 0.09236808674864493

Performance:

Dataset Speed RAM
LiDiRus - -
RCB - -
PARus - -
MuSeRC - -
TERRa - -
RUSSE - -
RWSD - -
DaNetQA - -
RuCoS - -