On the Importance and Risks in Using Sub-Reviewers
I promised to post from time to time about the FOCS 2013 PC work (to demystify the process). So here is a quick update: We got 280 submission (well, 281 submissions but one was just a bad joke). This is up (by more than %10) from FOCS 2012 but on the other hand our PC is also on the large side so the number of papers assigned to a PC member is less than 40, which for FOCS/STOC is very reasonable (I reduced this number a bit by encouraging several authors of misplaced papers to withdraw their submissions).
The first stage of the work was assigning each paper to three PC members (more challenging than I expected), and then the individual review stage of the PC work. In this stage, each PC member reviews his/her assigned papers without discussion with other PC members. This way, each paper gets three relatively independent reviews. We are now ending this stage, and it is a good opportunity to talk about using sub-reviewers.
A PC member can ask an expert in the specific sub-area of a paper for a review (and we then refer to such an expert as a sub-reviewer). The purpose of sub-reviewers is not to lighten the load of PC members (though this could be seen as a positive byproduct), but rather to elicit expertise the PC is missing. One disadvantage of sub-reviewers which I will not consider here is the additional work for the rest of the community. This is a consideration, but hopefully an expert would be interested in reviewing papers that are very close to his/her expertise (and this would not be too much work).
It is my experience that a major source of mistakes for PCs is in the way they use external experts. One fear is an over-confident PC that believes it can judge every work on its own. True enough “in the good old days,” PCs were self-sufficient (especially in the pre-Internet era). But TOC today is much deeper and much wider than “in the good old days.” It is easy to make mistakes if you do not have good understanding of related work. On the other hand, PCs that rely on their sub-reviewers too much are in just a serious problem. The risk is to turn FOCS/STOC into a collection of mini-conferences run by separate sub-communities. This is completely opposite to the purpose of FOCS/STOC: promoting TOC as a joint community and fostering the flow of ideas. For that to happen we must have that papers accepted are of interest to larger fractions of the community. The ideal solution in my eye is to heavily rely on external experts, but to view these reviews as inputs to the PC work rather than as outsourcing of the PC work. I therefore asked each PC member to familiarize himself/herself with its assigned papers and to reach a separate opinion. I know this is an ideal view, but setting the right ideals could be important.