You are heading to Paris for the first time. You finally get there and it is time for dinner, you need to pick a restaurant. Of course you would like to avoid Tourists-Traps. Traditionally, people were relying on rule of thumb, and if they were lucky and knew someone that has some useful information, they could collect recommendations and use those in their decisions. Today, we rely heavily on information from recommendations of previous costumers (mostly complete strangers), for example, when using sites like Yelp to make such decisions. Such sites have “changed the game” for the providers: from a one shot interaction with individual costumers, in which the future cost of providing low quality food for high price is negligible (due to a stream of new and naïve consumers that keep coming, unaware of the experience of prior costumers) to a repeated interaction with a community of users in which good service is rewarded by future business. The existence of mechanisms for information sharing within the community creates an incentive for restaurants to improve and provide better service.
Now, consider the system we are now using to review papers to conferences. All top conferences have acceptance rate well below 50%, meaning that the majority of papers are rejected. Many (if not most) of these papers will be submitted to later conferences, sometimes even the same conference in the following year. Each time a paper is submitted to a conference it (usually) gets a fresh set of reviewers, completely unaware of prior submissions of that paper to previous conferences, and to the comments it got in previous reviews. In many cases these previous reviews point out issues that are important and yet sometimes overlooked by the new reviewers. Such a loss is clearly inefficient, and the current system creates an incentive to submit rejected papers, sometimes without properly addressing issues raised by reviewers, in order to get a “fresh roll of the dice”.
Reviewing papers is a major community effort and as a community we should make sure that effort is not wasted. The huge burden of reviews also results with less time to invest in each review, and as a result quality of reviews decreases and the community suffers. We would like to propose the following system for discussion:
- A conference that chooses to participate in the system will declare at the CFP that it expects that authors of each paper that is submitted to attach a document that includes a declaration of all prior submissions of the paper to conferences, the reviews that have been received and the way the authors addressed the criticism in these reviews. Authors are definitely allowed not to accept every comment and make changes accordingly, but they should explain the reasons for their decisions in the document.
- PC members and reviewers for the conference will not have access to the document before they submit their fresh review for the paper. This is to make sure their evaluation is not biased by previous reviews. After they submit their reviews, the PC members get access to the document and can check that they do not find any reasons there to change their review. If they do, they would submit a revision of their review (yet the PC chair would have access to both versions of the reviews so she could make the final call using the reviews as submitted before the access to the older reviews was granted).
Improvement in quality and reduce load of reviewing will not be a result of the program committee relying on the old reviews and saving the time to review the paper again, but rather due to indirect effects of the new system:
- The new system will give authors the incentive to self-select papers more carefully before resubmitting, and thus decrease the overall burden of reviews. Additionally, papers that are resubmitted will be of higher quality as authors will have a stronger incentive to address the issues raised in past reviews.
- Making prior reviews viewable to current PC members will also allow prior reviews to be evaluated over time and with the hindsight of authors response. Knowing that this will happen may indirectly increase the quality of reviews. In fact, in some communities (for example in statistics) public reviews and rebuttals are published along with the paper (but this is a matter for a later blog post).
This proposal is in the spirit of what is already happening at some extent when submitting a paper to a journal. Essentially, the current system we have in place to handle revision within the same journal (but not across journals) is similar to our proposal. When revising the paper after getting the first round of comments, authors are obligated to address the comments of the reviewers in a document. In journal reviews however, the reviewers of the revision are usually the same reviewers that reviewed the original submission, while that will probably not be the case when submitting to a different conference.
Moshe and Ittai