June 4, 2020, 3:26 p.m.
Team: AGI NLP
Model url: https://github.com/google-research/bert/blob/master/multilingual.md
| Dataset | Score | Metric |
|---|---|---|
| LiDiRus | 0.157 | Matthew`s Corr |
| RCB | 0.425 / 0.365 | F1/Acc |
| PARus | 0.588 | Accuracy |
| MuSeRC | 0.626 / 0.253 | F1a/Em |
| TERRa | 0.62 | Accuracy |
| RUSSE | 0.84 | Accuracy |
| RWSD | 0.675 | Accuracy |
| DaNetQA | 0.79 | Accuracy |
| RuCoS | 0.371 / 0.367 | F1/EM |
MultilingualBERT, version `bert-base-multilingual-cased` in transformers
Pre-trained for every task with early stopping and limited number of epochs
| Category | Score |
|---|---|
| LOGIC | 0.0996418167865508 |
| KNOWLEDGE | 0.109369990486137 |
| PREDICATE-ARGUMENT STRUCTURE | 0.169313385486666 |
| LEXICAL SEMANTICS | 0.192048601923186 |
| Lexical Semantics - Lexical Entailment | 0.0197130644086332 |
|---|---|
| Lexical Semantics - Morphological Negation | 0.154303349962092 |
| Lexical Semantics - Factivity | 0.260289603147677 |
| Lexical Semantics - Symmetry/Collectivity | -0.121716123890037 |
| Lexical Semantics - Redundancy | 0.675663924692176 |
| Lexical Semantics - Named Entities | 0.0556414884074657 |
| Lexical Semantics - Quantifiers | 0.259717774326225 |
| Predicate-Argument Structure Core Args | 0.297107355542109 |
| Predicate-Argument Structure Prepositional Phrases | 0.346654828223026 |
| Predicate-Argument Structure Ellipsis/Implicits | 0.161374306091976 |
| Predicate-Argument Structure Anaphora/Coreference | -0.0266720016005335 |
| Predicate-Argument Structure Active/Passive | 0.324442842261525 |
| Predicate-Argument Structure Nominalization | 0.236316335931351 |
| Predicate-Argument Structure Genitives/Partitives | 0.375 |
| Predicate-Argument Structure Datives | 0.285112401149233 |
| Predicate-Argument Structure Relative Clauses | -0.0666666666666667 |
| Predicate-Argument Structure Coordination Scopes | 0.168345124585359 |
| Predicate-Argument Structure Intersectivity | 0.193795910949418 |
| Predicate-Argument Structure Restrictivity | 0.274873708374511 |
| Logic Negation | -0.0344536400792846 |
| Logic Double Negation | -0.0753778361444409 |
| Logic Interval/Numbers | 0.0103005240524921 |
| Logic Conjuction | 0.325395686727984 |
| Logic Disjunction | -0.0528680532637681 |
| Logic Conditionals | -0.107762358448765 |
| Logic Universal | 0.644658371220304 |
| Logic Existential | 0.251629955979423 |
| Logic Temporal | 0.0621027319149067 |
| Logic Upward Monotone | 0.361234352275241 |
| Logic Downward Monotone | 0.269069117598525 |
| Logic Non-Monotonic | -0.168505092748454 |
| Knowledge Common Sense | 0.0239437075024363 |
| Knowledge World Knowledge | 0.186694873255926 |