Task TERRa

Name Identifier Type of the task Metrics License Download HB score Baseline score
Textual Entailment Recognition for Russian TERRa Binary Classification Accuracy MIT License 0.92 0.62

Description

Textual Entailment Recognition has been proposed recently as a generic task that captures major semantic inference needs across many NLP applications, such as Question Answering, Information Retrieval, Information Extraction, and Text Summarization. This task requires to recognize, given two text fragments, whether the meaning of one text is entailed (can be inferred) from the other text.

Task Type

RTE (Recognizing Textual Entailment) Sentence Pair Classification - Entailment - Not Entailment

Example


{
  "premise": "Автор поста написал в комментарии, что прорвалась канализация.",
  "hypothesis": "Автор поста написал про канализацию.",
  "label": "entailment",
  "idx": "6062"
}
        

How did we collect data?

All text examples were collected from open news sources and literary magazines, then manually reviewed and supplemented by a human assessment on Yandex.Toloka

RTE Challanges (ACL)

State of the Art

English RTE - Accuracy 92.5%