Task LiDiRus

Name Identifier Type of the task Metrics License Download HB score
Diagnostic LiDiRus Classification Matthews Corr MIT License 0.626

Description

LiDiRus is a diagnostic dataset that covers a large volume of linguistic phenomena, while allowing you to evaluate information systems on a simple test of textual entailment recognition. See more details diagnostics.

Task Type

RTE (Recognizing Textual Entailment) Sentence Pair Classification - Entailment - Not Entailment

Example

{
     'sentence1': "Кошка сидела на коврике.",
     'sentence2': "Кошка не сидела на коврике.",
     'label': 'not_entailment',
     'knowledge': '',
     'lexical-semantics': '',
     'logic': 'Negation',
     'predicate-argument-structure': ''
    }

How did we collect data?

All text examples manually translated and adapted from English SuperGLUE Diagnostics

See also:

RTE Challanges (ACL)

State of the Art

English Diagnostics - Matthew`s Corr 65.6%

Related papers