Feb. 8, 2021, 4:58 p.m.
Team: hse_ling
Dataset | Score | Metric |
---|---|---|
LiDiRus | 0.147 | Matthew`s Corr |
RCB | 0.4 / 0.438 | F1/Acc |
PARus | 0.478 | Accuracy |
MuSeRC | 0.671 / 0.237 | F1a/Em |
TERRa | 0.549 | Accuracy |
RUSSE | 0.595 | Accuracy |
RWSD | 0.669 | Accuracy |
DaNetQA | 0.642 | Accuracy |
RuCoS | 0.26 / 0.257 | F1/EM |
Simple rule-based heuristics like: full match of the instance: (RCB) premise: "Из материалов дела следует, что начальник одного из отделов пытался выбить субсидию заинтересованной организации за «откат»." hypothesis: "Начальник одного из отделов пытался выбить субсидию заинтересованной организации за «откат»." idx (training data): 12 instance's length: (TERRa) if premise has less than 29 words in it's likely that the label is "not_entailment" same form: (RUSSE) if word forms are the same it is likely that the label is "False". For instances where an heurstic does not work, the label was predicted with the help of majority class (based on training and validation datasets combined)
Category | Score |
---|---|
LOGIC | 0.20234318712510463 |
KNOWLEDGE | 0.11732351833491794 |
PREDICATE-ARGUMENT STRUCTURE | 0.13699734914800707 |
LEXICAL SEMANTICS | 0.17542964532360103 |
Lexical Semantics - Lexical Entailment | 0.1431860686204434 |
---|---|
Lexical Semantics - Morphological Negation | 0.14085904245475275 |
Lexical Semantics - Factivity | 0.31622776601683794 |
Lexical Semantics - Symmetry/Collectivity | -0.41079191812887456 |
Lexical Semantics - Redundancy | 0.2211629342323457 |
Lexical Semantics - Named Entities | 0.0 |
Lexical Semantics - Quantifiers | 0.3295017884191656 |
Predicate-Argument Structure Core Args | -0.11013775835078694 |
Predicate-Argument Structure Prepositional Phrases | 0.37605071654517747 |
Predicate-Argument Structure Ellipsis/Implicits | 0.16735967034880075 |
Predicate-Argument Structure Anaphora/Coreference | 0.05945643676931176 |
Predicate-Argument Structure Active/Passive | -0.22213082915965962 |
Predicate-Argument Structure Nominalization | 0.1414213562373095 |
Predicate-Argument Structure Genitives/Partitives | 0.16666666666666666 |
Predicate-Argument Structure Datives | 0.3273268353539886 |
Predicate-Argument Structure Relative Clauses | -0.09759000729485333 |
Predicate-Argument Structure Coordination Scopes | 0.3126409456585232 |
Predicate-Argument Structure Intersectivity | 0.21119243915593267 |
Predicate-Argument Structure Restrictivity | 0.2627807231132027 |
Logic Negation | 0.16891891891891891 |
Logic Double Negation | 0.3302891295379082 |
Logic Interval/Numbers | -0.13130643285972254 |
Logic Conjuction | 0.08606629658238704 |
Logic Disjunction | 0.29186501192363773 |
Logic Conditionals | 0.32277655457716287 |
Logic Universal | 0.08058229640253803 |
Logic Existential | 0.10482848367219183 |
Logic Temporal | 0.07984636960487998 |
Logic Upward Monotone | 0.21022325722438134 |
Logic Downward Monotone | 0.035955873264830976 |
Logic Non-Monotonic | 0.3152441624956403 |
Knowledge Common Sense | 0.15188811433521146 |
Knowledge World Knowledge | 0.09236808674864493 |
Dataset | Speed | RAM |
---|---|---|
LiDiRus | - | - |
RCB | - | - |
PARus | - | - |
MuSeRC | - | - |
TERRa | - | - |
RUSSE | - | - |
RWSD | - | - |
DaNetQA | - | - |
RuCoS | - | - |