← All publications

Can NLP Models Correctly Reason Over Contexts That Break the Common Assumptions?

Neeraj Varshney, Mihir Parmar, Nisarg Patel, Divij Handa, Sayantan Sarkar, Man Luo, Chitta Baral

Preprint 2023 Evaluation & Safety
arXiv ↗

Abstract

Systematically constructs contexts that break common assumptions and shows that, while models reason well over assumption-following contexts, performance drops by up to 20% when those assumptions are broken.

Reasoning