Michael Mitzenmacher points to two posts of Suresh Venkatasubramanian on the issue of so called “double blind reviews” (i.e., anonymous submissions) in theory conferences. In short, both Michael and Suresh think they are a good idea. I agree with much of their motivations, but, based on my experience in both non-blinded (e.g., STOC/FOCS) and blinded (e.g., CRYPTO) conferences as both reviewer and author, I do not think double blind reviewing is a good fit for theoretical computer science.
Let me say right off the bat that I think implicit (and, as Michael says, sometimes explicit) bias is a very real phenomenon. Moreover, such biases are not just a problem in the sense that they are “unfair” to authors, but they cause real harm to science, in suppressing the contributions from certain authors. Nor do I have any principled objection to anonymization: I do for example practice anonymous grading in my courses for exactly this reason. I also don’t buy the suggestion that we must know the author’s identity to evaluate if the proof is correct. Reviewers can (and do) evaluate whether a proof makes sense without needing to trust the author.
However, there is a huge difference between grading a problem set and refereeing a paper. In the latter case, and in particular in theoretical computer science, you often need the expertise of very particular people that have worked on this area. By the time the paper is submitted to a conference, these experts have often already seen it, either because it was posted on the arxiv/eccc/eprint, or because they have seen a talk on it, or perhaps they have already discussed it with the authors by email.
More generally, these days much of theoretical CS is moving to the model where papers are first posted online, and by the time they are submitted to a conference they have circulated quite a bit around the relevant experts. Posting papers online is very good for science and should be encouraged, as it allows fast dissemination of results, but it does make the anonymous submission model obsolete.
One could say that if the author’s identity is revealed then there is no harm, since in such a case we simply revert to the original form of non anonymous submissions. However, the fact that the authors’ identity is known to some but not all participants in the process (e.g., maybe some reviewers but not others), makes some conflicts and biases invisible. Moreover, the fact that the author’s identity is not “officially” known, causes a lot of practical headaches.
For example, as a PC member you can’t just shoot a quick email to an expert to ask for a quick opinion on the paper, since they may well be the author themselves (as happened to me several time as a CRYPTO PC member), or someone closely related to them. Second, you often have the case where the reviewer knows who the authors are, and has some history with them, even if it’s not a formal conflict, but the program committee member does not know this information. In particular, using anonymous submissions completely precludes using a disclosure based model for conflicts of interest (where reviewers disclose their relations with the authors in their reviews) but rather you have to move to an exclusion based model, where reviewers meeting some explicit criteria are ruled out.
If anonymous submissions don’t work well for theory conferences, does it mean we have to just have to accept biases? I don’t think so. I believe there are a number of things we could attempt. First, while completely anonymizing submissions might not work well, we could try to make the author names less prominent, for example by having them in the last page of the submissions instead of the first, and not showing them in the conference software. Also, we could try “fairness through awareness”. As I mentioned in my tips for future FOCS/STOC chairs, one potential approach is to tag papers by authors who never had a prior STOC/FOCS paper (one could possibly also tag papers by authors from under-represented groups). One wouldn’t give such papers preferential treatment, but rather just make sure they get extra attention. For example, we could add an extra review for such papers. That review might end up being positive or negative, but would counter the bias of dismissing some works out of hand.
To summarize, I agree with Michael’s and Suresh’s sentiments that biases are harmful and should be combated. I just don’t think anonymous submissions are the way to go about that.
Windows On Theory 
“I also don’t buy the suggestion that we must know the author’s identity to evaluate if the proof is correct.”
I will make the same objection here that I made before: One need not trivialize the argument in order to argue for or against it. Of course it’s not necessary to know the authors’ identity to verify correctness. That’s the whole idea of a proof. But you would not have posted about the 2->2 conjecture so quickly (it is a long, technical, and still not independently verified) if the authors had been three unknown students. The approach (expansion in the Grassman graphs) itself would have given you some confidence, but I suspect you would still be more cautious.
I worry more about systemic bias than implicit bias (it’s easier to worry about). People tend to think of villains (those selfish people who can’t avoid pushing the needle in their own favor). But the model of non-anonymous reviewing creates a situation in which the social optimum is unfair, even though every reviewer is acting in a fair and rational way.
As the chair of graduate admissions right now, this phenomenon is particularly clear. We have 2000+ applications, but a limited pool of humans able to read and carefully evaluate them. So we allocate attention based on priors, and those priors are determined largely by the undergraduate institution of the applicant. (It is hard to argue that this isn’t the signal that maximizes the ratio of its value to the effort to measure the signal.)
Every application has at least one human reader and outstanding applications are assigned more readers. But the end result is probably the following: The very best applicants get attention, regardless of their institution. In the middle ground, where meaningfully distinguishing among applicants is harder, the process tends to favor people from well-known institutions simply because one can get a final pool of the same quality with comparably less expended effort.
With more resources, fairness would improve. Fairness is expensive. That doesn’t mean it isn’t worth it. (I personally think it is.) But I don’t think saying “Look, fairness is actually cheap” by dismissing the value of things (like knowing that the author is a domain expert) is a realistic approach.
Hi James,
I was perhaps too succinct. It’s not that in general, knowing the author does not save us a lot of time in adjusting our priors in our day to day lives. But I do think that (unlike the graduate admissions case) having the names of the authors is not a huge time saver for the sake of verifying correctness in conference refereeing.
This is because:
(a) For most reviewed papers, the proof verification part is not the time consuming part. Most papers are rejected because the result is not interesting, not because we suspect the proof is incorrect. And since we don’t do full verification but rather check that a proof makes sense, it is only some papers with particularly subtle proofs where we need to really delve deeply.
(b) That said, for some papers, it really all boils down to the proof, and you can’t get a good sense for it without investing significant effort. To the credit of the theory community, in such cases I’ve seen time and again referees invest significant time and effort in understanding the proof (in some cases having many rounds of back and forth with the authors in clarifying proofs) even when the author is well known. We don’t just blindly trust the author. In particular, while you’re right that I posted about the 2 to 2 conjecture because I know the authors, I wouldn’t have done that if I had to review the paper for a conference. I would not go over the proof line by line, but would make sure I understand its structure and that it all makes sense, whether it is by these guys or an author I never heard about before.
Fairness has some cost, but there might be some good cost/benefit tradeoffs. In particular I do like Omer’s suggestion of simply putting the authors at the end of the paper.
Thanks Boaz.
(I don’t think anyone ever advocated for “blind trust” in the authors…)
I agree that, once you have decided to verify the correctness of a proof, the names of the authors is much less important. In that sense, the ‘attention allocation’ problem falls back to determining which results are interesting and which aren’t, and I suppose that issue exists in every field.
Should we anonymize NSF panels? NSF proposals are usually pretty speculative. The credibility of the PI in being able to carry out the proposed work is often a central issue for the panel (I think even explicitly discussed in the instructions).
I don’t think there is a “one size fits all” approach, and (as I said) I actually agree with you that even for paper reviewing we shouldn’t use anonymous submissions.
There is absolutely no sense in anonymizing NSF panels, indeed the credibility of the PI is absolutely crucial there, since it’s looking at the future, and the way to predict it is based on the past. Similarly, we probably shouldn’t anonymize faculty hiring 🙂 (though if we wanted to do so, we really should insist they wear masks until they get tenure.. )
The NSF is very clear that their judgement of a proposal is based on favorable answers to three questions:
* Is the problem worth studying
* is the approach described likely to yield favorable results
* is the team proposing the approach likely to execute
You can’t answer the last part without author identities.
I haven’t heard anyone mention the following natural idea, which I’ll take the opportunity to name “sesqui-blind reviewing”:
The idea is simply to let the PC know the authors’ names/affiliations so that they can avoid conflicted subreviewers, but not to share the authors’ names/affiliations with the subreviewers themselves. One can also consider a variation of this in which PC members see authors’ names/affiliations only if they explicitly choose to.
I think the benefits of this are obvious. Compared to the current system, it should greatly reduce the bias of subreviewers, but conflicts should be much easier to avoid.
COLT has used a system like this for the past three or four years. An email gets sent to the PC members with author names, but the names are not visible on easychair and on the paper. The system works reasonably well for the community — some reviewers can guess who the authors are, most can’t and many guess incorrectly (I sometimes ask!)
The bigger ML conferences like ICML, NIPS and AISTATS have two level program committees; the senior PC knows the author names and makes reviewer assignments from a predefined pool (containing almost everybody in the community). Again I think the system works as well as can be expected; there’s a fair bit of noise in the system, but my impression is that the noise is largely uncorrelated with author names. If anyone has any advantage, it tends to be the authors who write lucid and polished papers.
Actually this model has been suggested, most notably by Kathryn McKinley in her article on best practices with double blind review: the idea is that the first-line reviewers are the ones who don’t see the author names, but that author names can be progressively revealed at higher levels of the review process. Since the point of blinding is to try and eliminate the initial subconscious judgements, this works reasonably well.
I’m not sure I’m that convinced by the idea of pushing author names to the back of the paper. It will then become a matter of course to just look at the back page.