12 ноября 2020 г. 11:57
Команда AGI NLP
Ссылка на модель https://github.com/RussianNLP/RussianSuperGLUE
Датасет | Результат | Метрика |
---|---|---|
LiDiRus | 0,626 | Кор, коэффициент Мэтью |
RCB | 0,68 / 0,702 | F1/Точность |
PARus | 0,982 | Точность |
MuSeRC | 0,806 / 0,42 | F1a/Em |
TERRa | 0,92 | Точность |
RUSSE | 0,805 | Точность |
RWSD | 0,84 | Точность |
DaNetQA | 0,915 | Точность |
RuCoS | 0,93 / 0,89 | F1/EM |
HUMAN BENCHMARK for version 1.2
Категория | Результат |
---|---|
LOGIC | |
KNOWLEDGE | |
PREDICATE-ARGUMENT STRUCTURE | |
LEXICAL SEMANTICS |
Lexical Semantics - Lexical Entailment | |
---|---|
Lexical Semantics - Morphological Negation | |
Lexical Semantics - Factivity | |
Lexical Semantics - Symmetry/Collectivity | |
Lexical Semantics - Redundancy | |
Lexical Semantics - Named Entities | |
Lexical Semantics - Quantifiers | |
Predicate-Argument Structure Core Args | |
Predicate-Argument Structure Prepositional Phrases | |
Predicate-Argument Structure Ellipsis/Implicits | |
Predicate-Argument Structure Anaphora/Coreference | |
Predicate-Argument Structure Active/Passive | |
Predicate-Argument Structure Nominalization | |
Predicate-Argument Structure Genitives/Partitives | |
Predicate-Argument Structure Datives | |
Predicate-Argument Structure Relative Clauses | |
Predicate-Argument Structure Coordination Scopes | |
Predicate-Argument Structure Intersectivity | |
Predicate-Argument Structure Restrictivity | |
Logic Negation | |
Logic Double Negation | |
Logic Interval/Numbers | |
Logic Conjuction | |
Logic Disjunction | |
Logic Conditionals | |
Logic Universal | |
Logic Existential | |
Logic Temporal | |
Logic Upward Monotone | |
Logic Downward Monotone | |
Logic Non-Monotonic | |
Knowledge Common Sense | |
Knowledge World Knowledge |
Датасет | Speed | RAM |
---|---|---|
LiDiRus | - | - |
RCB | - | - |
PARus | - | - |
MuSeRC | - | - |
TERRa | - | - |
RUSSE | - | - |
RWSD | - | - |
DaNetQA | - | - |
RuCoS | - | - |