Aug. 18, 2021, 11:55 a.m.
Team: SberDevices
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.235 | Matthew`s Corr |
| RCB | 0.356 / 0.5 | F1/Acc |
| PARus | 0.656 | Accuracy |
| MuSeRC | 0.778 / 0.436 | F1a/Em |
| TERRa | 0.704 | Accuracy |
| RUSSE | 0.707 | Accuracy |
| RWSD | 0.669 | Accuracy |
| DaNetQA | 0.773 | Accuracy |
| RuCoS | 0.81 / 0.805 | F1/EM |
Russian Bert-large model. 430ml parameters, trained on 30 GB Russian texts.
-
| Category | Score |
|---|---|
| LOGIC | 0.13965594076204224 |
| KNOWLEDGE | 0.25420452490107115 |
| PREDICATE-ARGUMENT STRUCTURE | 0.2110557307641125 |
| LEXICAL SEMANTICS | 0.23178921890059834 |
| Lexical Semantics - Lexical Entailment | 0.1605357781060343 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.060993754559283325 |
| Lexical Semantics - Factivity | 0.21320071635561044 |
| Lexical Semantics - Symmetry/Collectivity | 0.0 |
| Lexical Semantics - Redundancy | 0.45652173913043476 |
| Lexical Semantics - Named Entities | 0.1767766952966369 |
| Lexical Semantics - Quantifiers | 0.1913754029921869 |
| Predicate-Argument Structure Core Args | 0.33895960971961925 |
| Predicate-Argument Structure Prepositional Phrases | 0.44921394657019476 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.14818474149146235 |
| Predicate-Argument Structure Anaphora/Coreference | 0.07001400420140048 |
| Predicate-Argument Structure Active/Passive | 0.3059179896521843 |
| Predicate-Argument Structure Nominalization | 0.0 |
| Predicate-Argument Structure Genitives/Partitives | 0.25 |
| Predicate-Argument Structure Datives | 0.37796447300922725 |
| Predicate-Argument Structure Relative Clauses | 0.13912166872805048 |
| Predicate-Argument Structure Coordination Scopes | 0.20894746626614943 |
| Predicate-Argument Structure Intersectivity | 0.03545133231866123 |
| Predicate-Argument Structure Restrictivity | -0.1455213750217998 |
| Logic Negation | 0.08219949365267865 |
| Logic Double Negation | 0.0 |
| Logic Interval/Numbers | 0.2876779808912305 |
| Logic Conjuction | 0.29370130064810657 |
| Logic Disjunction | 0.2780367699071248 |
| Logic Conditionals | 0.14285714285714285 |
| Logic Universal | 0.5324675324675324 |
| Logic Existential | 0.10482848367219183 |
| Logic Temporal | -0.12434118282549846 |
| Logic Upward Monotone | 0.31353483628976775 |
| Logic Downward Monotone | -0.09513029883089882 |
| Logic Non-Monotonic | 0.0 |
| Knowledge Common Sense | 0.3022898014232672 |
| Knowledge World Knowledge | 0.17129218721431935 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | - | - |
| RCB | - | - |
| PARus | - | - |
| MuSeRC | - | - |
| TERRa | - | - |
| RUSSE | - | - |
| RWSD | - | - |
| DaNetQA | - | - |
| RuCoS | - | - |