4 июня 2020 г. 15:26
Команда AGI NLP
Ссылка на модель https://github.com/google-research/bert/blob/master/multilingual.md
| Датасет | Результат | Метрика |
|---|---|---|
| LiDiRus | 0,157 | Кор, коэффициент Мэтью |
| RCB | 0,425 / 0,365 | F1/Точность |
| PARus | 0,588 | Точность |
| MuSeRC | 0,626 / 0,253 | F1a/Em |
| TERRa | 0,62 | Точность |
| RUSSE | 0,84 | Точность |
| RWSD | 0,675 | Точность |
| DaNetQA | 0,79 | Точность |
| RuCoS | 0,371 / 0,367 | F1/EM |
MultilingualBERT, version `bert-base-multilingual-cased` in transformers
Pre-trained for every task with early stopping and limited number of epochs
| Категория | Результат |
|---|---|
| LOGIC | 0,0996418167865508 |
| KNOWLEDGE | 0,109369990486137 |
| PREDICATE-ARGUMENT STRUCTURE | 0,169313385486666 |
| LEXICAL SEMANTICS | 0,192048601923186 |
| Lexical Semantics - Lexical Entailment | 0,0197130644086332 |
|---|---|
| Lexical Semantics - Morphological Negation | 0,154303349962092 |
| Lexical Semantics - Factivity | 0,260289603147677 |
| Lexical Semantics - Symmetry/Collectivity | -0,121716123890037 |
| Lexical Semantics - Redundancy | 0,675663924692176 |
| Lexical Semantics - Named Entities | 0,0556414884074657 |
| Lexical Semantics - Quantifiers | 0,259717774326225 |
| Predicate-Argument Structure Core Args | 0,297107355542109 |
| Predicate-Argument Structure Prepositional Phrases | 0,346654828223026 |
| Predicate-Argument Structure Ellipsis/Implicits | 0,161374306091976 |
| Predicate-Argument Structure Anaphora/Coreference | -0,0266720016005335 |
| Predicate-Argument Structure Active/Passive | 0,324442842261525 |
| Predicate-Argument Structure Nominalization | 0,236316335931351 |
| Predicate-Argument Structure Genitives/Partitives | 0,375 |
| Predicate-Argument Structure Datives | 0,285112401149233 |
| Predicate-Argument Structure Relative Clauses | -0,0666666666666667 |
| Predicate-Argument Structure Coordination Scopes | 0,168345124585359 |
| Predicate-Argument Structure Intersectivity | 0,193795910949418 |
| Predicate-Argument Structure Restrictivity | 0,274873708374511 |
| Logic Negation | -0,0344536400792846 |
| Logic Double Negation | -0,0753778361444409 |
| Logic Interval/Numbers | 0,0103005240524921 |
| Logic Conjuction | 0,325395686727984 |
| Logic Disjunction | -0,0528680532637681 |
| Logic Conditionals | -0,107762358448765 |
| Logic Universal | 0,644658371220304 |
| Logic Existential | 0,251629955979423 |
| Logic Temporal | 0,0621027319149067 |
| Logic Upward Monotone | 0,361234352275241 |
| Logic Downward Monotone | 0,269069117598525 |
| Logic Non-Monotonic | -0,168505092748454 |
| Knowledge Common Sense | 0,0239437075024363 |
| Knowledge World Knowledge | 0,186694873255926 |