Nov. 18, 2020, 7:43 p.m.
Team: DeepPavlov
Model url: https://huggingface.co/bert-base-multilingual-cased
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.189 | Matthew`s Corr |
| RCB | 0.367 / 0.445 | F1/Acc |
| PARus | 0.528 | Accuracy |
| MuSeRC | 0.639 / 0.239 | F1a/Em |
| TERRa | 0.617 | Accuracy |
| RUSSE | 0.69 | Accuracy |
| RWSD | 0.669 | Accuracy |
| DaNetQA | 0.624 | Accuracy |
| RuCoS | 0.29 / 0.29 | F1/EM |
bert-base-multilingual-cased
Standard
| Category | Score |
|---|---|
| LOGIC | 0.13735664681948273 |
| KNOWLEDGE | 0.1265469818958828 |
| PREDICATE-ARGUMENT STRUCTURE | 0.18920483573043412 |
| LEXICAL SEMANTICS | 0.22242373756652029 |
| Lexical Semantics - Lexical Entailment | 0.0584543445233507 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.3003124875651863 |
| Lexical Semantics - Factivity | 0.3266793038882825 |
| Lexical Semantics - Symmetry/Collectivity | -0.12171612389003691 |
| Lexical Semantics - Redundancy | 0.45652173913043476 |
| Lexical Semantics - Named Entities | 0.055641488407465724 |
| Lexical Semantics - Quantifiers | 0.2597177743262246 |
| Predicate-Argument Structure Core Args | 0.2297707831614211 |
| Predicate-Argument Structure Prepositional Phrases | 0.3972493441698867 |
| Predicate-Argument Structure Ellipsis/Implicits | -0.014731391274719742 |
| Predicate-Argument Structure Anaphora/Coreference | 0.23337331720804128 |
| Predicate-Argument Structure Active/Passive | 0.2559239653166382 |
| Predicate-Argument Structure Nominalization | -0.025949964805384102 |
| Predicate-Argument Structure Genitives/Partitives | 0.15724272550828775 |
| Predicate-Argument Structure Datives | 0.2182178902359924 |
| Predicate-Argument Structure Relative Clauses | 0.3333333333333333 |
| Predicate-Argument Structure Coordination Scopes | 0.20894746626614943 |
| Predicate-Argument Structure Intersectivity | 0.23234812346321748 |
| Predicate-Argument Structure Restrictivity | 0.0 |
| Logic Negation | 0.040398847130309966 |
| Logic Double Negation | 0.0 |
| Logic Interval/Numbers | 0.03262334552859464 |
| Logic Conjuction | 0.4173650061841514 |
| Logic Disjunction | 0.17035261607214167 |
| Logic Conditionals | -0.24039295346263034 |
| Logic Universal | 0.5606119105813882 |
| Logic Existential | -0.04279604925109129 |
| Logic Temporal | 0.11396057645963795 |
| Logic Upward Monotone | 0.4083133966424866 |
| Logic Downward Monotone | 0.2788995520575487 |
| Logic Non-Monotonic | 0.10244389988519648 |
| Knowledge Common Sense | 0.07314923469853085 |
| Knowledge World Knowledge | 0.15662622780364913 |
| Dataset | Speed | RAM |
|---|---|---|
| LiDiRus | 136 | 2.39 |
| RCB | 194 | 2.39 |
| PARus | 451 | 2.39 |
| MuSeRC | 4 | 2.39 |
| TERRa | 195 | 2.39 |
| RUSSE | 164 | 2.39 |
| RWSD | 85 | 2.40 |
| DaNetQA | 90 | 2.40 |
| RuCoS | 7 | 2.40 |