Does My Rebuttal Matter? Insights from a Major NLP Conference. / Gao, Yang; Eger, Steffen; Kuznetsov, Ilia; Gurevych, Iryna; Miyao, Yusuke.

Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Vol. 1 Minneapolis, Minnesota : Association for Computational Linguistics, 2019. p. 1274–1290.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Published
  • Yang Gao
  • Steffen Eger
  • Ilia Kuznetsov
  • Iryna Gurevych
  • Yusuke Miyao

Abstract

Peer review is a core element of the scientific process, particularly in conference-centered fields such as ML and NLP. However, only few studies have evaluated its properties empirically. Aiming to fill this gap, we present a corpus that contains over 4k reviews and 1.2k author responses from ACL-2018. We quantitatively and qualitatively assess the corpus. This includes a pilot study on paper weaknesses given by reviewers and on quality of author responses. We then focus on the role of the rebuttal phase, and propose a novel task to predict after-rebuttal (i.e., final) scores from initial reviews and author responses. Although author responses do have a marginal (and statistically significant) influence on the final scores, especially for borderline papers, our results suggest that a reviewer’s final score is largely determined by her initial score and the distance to the other reviewers’ initial scores. In this context, we discuss the conformity bias inherent to peer reviewing, a bias that has largely been overlooked in previous research. We hope our analyses will help better assess the usefulness of the rebuttal phase in NLP conferences.
Original languageEnglish
Title of host publicationProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Place of PublicationMinneapolis, Minnesota
PublisherAssociation for Computational Linguistics
Pages1274–1290
Number of pages17
Volume1
Publication statusPublished - Jun 2019
This open access research output is licenced under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

ID: 34292231