Description

Russian Winograd Schema Dataset (RWSD), or the Winograd schema, is a task in which each example contains a sentence with two selected phrases. The task is to define whether they are used in the same sense or not. The schema takes its name from a well-known example by Terry Winograd.

The set would then be presented as a challenge for AI programs, along the lines of the Turing test. The strengths of the challenge are that it is clear-cut, in that the answer to each schema is a binary choice; vivid, in that it is obvious to non-experts that a program that fails to get the right answers clearly has serious gaps in its understanding; and difficult, in that it is far beyond the current state of the art.

Task Type¶

Logic and Reasoning, World knowledge. Binary Classification: true/false

Example¶

{
  "text": "Кубок не помещается в коричневый чемодан, потому что он слишком большой."
  "label": false,
  "idx": 5,
  "target": {
    "span1_text": "чемодан",
    "span2_text": "он слишком большой",
    "span1_index": 5,
    "span2_index": 8
  },
}

How did we collect data? ¶

All text examples were collected manually translating and adapting original Winograd dataset for Russian. Human assessment was carried out on Yandex.Toloka.

State of the Art

English Winograd Schema Challenge - Accuracy 93.8%

Related papers

• David Bender, Establishing a Human Baseline for the Winograd Schema Challenge. MAICS 2015
• Ernest Davis, Winograd Schemas and Machine Translation, arXiv:1608.01884, August 2016.
• Hector Levesque, The Winograd Schema Challenge, Commonsense-2011.
• Hector Levesque, Ernest Davis, and Leora Morgenstern, The Winograd Schema Challenge, KR-2012. An expanded version of the previous item.
• Hector Levesque, On Our Best Behaviour, IJCAI Research Excellence Award Presentation, 2013.
• Leora Morgenstern, E. Davis, and Charles Ortiz, Planning, Executing, and Evaluating the Winograd Schema Challenge, AI Magazine Spring 2016.
• Wang A. et al. Superglue: A stickier benchmark for general-purpose language understanding systems //Advances in Neural Information Processing Systems. – 2019. – С. 3261-3275.