Oct. 23, 2021, 1:03 p.m.
Team: Avengers Ensemble
Model url: https://github.com/neverix/avengers-ensemble/tree/v2
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.515 | Matthew`s Corr |
| RCB | 0.384 / 0.534 | F1/Acc |
| PARus | 0.906 | Accuracy |
| MuSeRC | 0.936 / 0.804 | F1a/Em |
| TERRa | 0.877 | Accuracy |
| RUSSE | 0.687 | Accuracy |
| RWSD | 0.643 | Accuracy |
| DaNetQA | 0.911 | Accuracy |
| RuCoS | 0.92 / 0.924 | F1/EM |
Решение - это ансамбль трансформеров. Тренируется много моделей, из них получаются вероятности и подаются в CatBoost. Один и тот же подход используется для всех задач, но больше всего моделей в RuCoS, DaNetQA, TERRA, MuSeRC и PARus. v2.0: Для RUSSE и RWSD натренированы дополнительные модели, некоторые из старых отключены. Для LiDiRus добавлено решение через zero-shot.
| Category | Score |
|---|---|
| LOGIC | 0.41692878129605887 |
| KNOWLEDGE | 0.41327428787001375 |
| PREDICATE-ARGUMENT STRUCTURE | 0.5098348436593766 |
| LEXICAL SEMANTICS | 0.5728036088438945 |
| Lexical Semantics - Lexical Entailment | 0.6199628384614737 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.6736330697086078 |
| Lexical Semantics - Factivity | 0.29499488356736703 |
| Lexical Semantics - Symmetry/Collectivity | 0.5087470190691683 |
| Lexical Semantics - Redundancy | 0.3448281541042982 |
| Lexical Semantics - Named Entities | 0.5698028822981898 |
| Lexical Semantics - Quantifiers | 0.6965723455474339 |
| Predicate-Argument Structure Core Args | 0.4782080487928343 |
| Predicate-Argument Structure Prepositional Phrases | 0.6524800478382362 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.5347222222222222 |
| Predicate-Argument Structure Anaphora/Coreference | 0.36300570155752676 |
| Predicate-Argument Structure Active/Passive | 0.6515837655350015 |
| Predicate-Argument Structure Nominalization | 0.8006407690254357 |
| Predicate-Argument Structure Genitives/Partitives | 0.7637626158259734 |
| Predicate-Argument Structure Datives | 0.6633880657639324 |
| Predicate-Argument Structure Relative Clauses | 0.42289003161103106 |
| Predicate-Argument Structure Coordination Scopes | 0.6308671104414854 |
| Predicate-Argument Structure Intersectivity | 0.27284292741384053 |
| Predicate-Argument Structure Restrictivity | 0.1503267973856209 |
| Logic Negation | 0.2529514557863603 |
| Logic Double Negation | 0.47100330099123966 |
| Logic Interval/Numbers | 0.12643678160919541 |
| Logic Conjuction | 0.7378647873726218 |
| Logic Disjunction | 0.40174192517093943 |
| Logic Conditionals | 0.5238095238095238 |
| Logic Universal | 0.6446583712203042 |
| Logic Existential | 0.2058790548922549 |
| Logic Temporal | 0.42276002160669474 |
| Logic Upward Monotone | 0.7636174600000242 |
| Logic Downward Monotone | -0.27865801780244004 |
| Logic Non-Monotonic | 0.15374072439330316 |
| Knowledge Common Sense | 0.500763612805955 |
| Knowledge World Knowledge | 0.31097697294880394 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |