April 25, 2024, 6:33 a.m.
Team: Maxim Bolgov
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.334 | Matthew`s Corr |
| RCB | 0.442 / 0.482 | F1/Acc |
| PARus | 0.61 | Accuracy |
| MuSeRC | 0.725 / 0.254 | F1a/Em |
| TERRa | 0.717 | Accuracy |
| RUSSE | 0.464 | Accuracy |
| RWSD | 0.695 | Accuracy |
| DaNetQA | 0.791 | Accuracy |
| RuCoS | 0.43 / 0.42 | F1/EM |
Qwen 14B by Alibaba, full parameter fine-tuned on Saiga dataset for 1 epoch.
| Category | Score |
|---|---|
| LOGIC | 0.26494440817298587 |
| KNOWLEDGE | 0.25632452508630893 |
| PREDICATE-ARGUMENT STRUCTURE | 0.32435006166584013 |
| LEXICAL SEMANTICS | 0.4632212533846751 |
| Lexical Semantics - Lexical Entailment | 0.3781169292734477 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.5476190476190477 |
| Lexical Semantics - Factivity | 0.408248290463863 |
| Lexical Semantics - Symmetry/Collectivity | 0.2921186973360886 |
| Lexical Semantics - Redundancy | 0.6922186552431729 |
| Lexical Semantics - Named Entities | 0.40482045237636816 |
| Lexical Semantics - Quantifiers | 0.4816727030991569 |
| Predicate-Argument Structure Core Args | 0.4778552908698884 |
| Predicate-Argument Structure Prepositional Phrases | 0.26323550158162423 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.225101084147303 |
| Predicate-Argument Structure Anaphora/Coreference | 0.169859653214886 |
| Predicate-Argument Structure Active/Passive | 0.45241392835886407 |
| Predicate-Argument Structure Nominalization | 0.5477225575051662 |
| Predicate-Argument Structure Genitives/Partitives | 0.6666666666666666 |
| Predicate-Argument Structure Datives | 0.641688947919748 |
| Predicate-Argument Structure Relative Clauses | 0.29277002188455997 |
| Predicate-Argument Structure Coordination Scopes | 0.1188643277625491 |
| Predicate-Argument Structure Intersectivity | 0.32489314482696546 |
| Predicate-Argument Structure Restrictivity | 0.0 |
| Logic Negation | 0.34363701612718517 |
| Logic Double Negation | 0.3481553119113957 |
| Logic Interval/Numbers | 0.010300524052492094 |
| Logic Conjuction | 0.20672455764868075 |
| Logic Disjunction | 0.36035409316916067 |
| Logic Conditionals | -0.050964719143762556 |
| Logic Universal | 0.3223291856101521 |
| Logic Existential | 0.48038446141526137 |
| Logic Temporal | 0.3829708431025352 |
| Logic Upward Monotone | 0.3244428422615251 |
| Logic Downward Monotone | 0.07098646750254821 |
| Logic Non-Monotonic | 0.2163856337073152 |
| Knowledge Common Sense | 0.2245231365137664 |
| Knowledge World Knowledge | 0.28078011281367116 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |