Sanjeev Arora: Thoughts on Paper Publishing in the Digital Age

In this guest post, Sanjeev Arora will share some thoughts about the future of scientific publishing in our community. This is not unrelated to our last post, and is also aimed at initiating discussion towards FOCS 2013 that is starting in the coming weekend.

As always, comments are most welcomed with the reminder that WindowsOnTheory maintains a policy of keeping the discussion respectful and on point. And now to Sanjeev:


Thoughts on paper publishing in the digital age.

What role should journals and conferences play in the age of arxiv, twitter and other yet-to-be-invented digital wonders? Detecting among many colleagues a general impatience with the status quo, I wrote this blog post to generate more public discussion. I am thinking here purely of theoretical computer science, not other fields.

It is possible now to envision a world where journals and conferences are replaced by arxiv and other depositories, which come with the added benefit of instant “impact metrics” (pageviews, number of tweets, etc.). Machine learning researcher Yann Le Cun has taken this viewpoint to its logical end, arguing for  a “free-market”system with papers subject to a distributed market model for refereeing/commenting. Independent consortia of reviewers (basically a new name for journals and conferences of the future??) would decide to publish reviews of arxiv-ed papers —with or without the permission of the papers’ authors.

At the other end of the spectrum, Lance Fortnow is troubled by the steady decline of journals and proliferation of conferences, and its bad effects on scholarship. He seeks to restore the importance of journals, and lower the prestige of conferences (by greatly raising acceptance rates).

Is Arxiv enough?


Physics is one of several fields that have taken enthusiastically to e-publishing. A physics paper on arxiv may have followup papers within weeks or even days.

While this model has some plus points, one also see dangers:(a) incentive to write shallow and incremental papers; (b) more priority disputes and the temptation to publish sketchy ideas in order to later claim full or at least partial credit; (c) lack of incentive for good reviewers to volunteer time reviewing enough papers (despite the attempted analogy to free market in Le Cun’s proposal).

Let me elaborate on (b). The following already seems to be an axiom among my younger colleagues: “If a result appears on arxiv, you have a few days to put up your independent manuscript. After that you can’t claim independent discovery.” Some other disciplines have already moved on to a more cutthroat model: “Whoever gets to arxiv first wins. “

Surely, this must incentivize hasty writing and incomprehensible papers. Or maybe papers that even have errors —but fixed in the weeks or months in subsequent versions. In the past we relied on conference committees and journals to adjudicate such disputes. How would that happen in a distributed market? One can imagine systems involving feedback buttons, reliability ratings etc. but it seems a dubious method of doing science.

An attraction of the “free market” approach at first sight is that unknown researchers can publish on equal footing with established ones. But in reality things may turn out less fair than the current conference system. Prominent researchers will have bigger megaphones (e.g., more twitter followers or friends willing to review their papers) and tend to benefit.  Power centers will inevitably form in any system.

By contrast, conferences in theoretical CS —perhaps because each PC is a fresh set of 25 individuals—have a good track record of showcasing great work by grad students and postdocs, and giving best paper awards to people I —and probably many others—had never heard of before. And plenty of Turing award winners get their papers rejected.

Conference vs Journals?

Historically, conferences came to dominate computer science because they allowed fast dissemination and a convenient place to catch up on the latest research/gossip. Today, both goals can increasingly be met by other means, so can we still justify conferences? It is interesting that Fortnow and Le Cun, despite being on opposite ends of the spectrum, agree about the irrelevance of conferences.

Let’s now list factors why conferences still make a lot of sense. My focus here is on promoting better science — I worry less about promotion/tenure policies since they will quickly adjust to accommodate any new dissemination method we choose (including arxiv and twitter). Also, I apologize in advance for occasional forays into pop psychology.

(a) Incentive system for good researchers to (sort-of )review lots of papers.

There simply isn’t enough refereeing capacity to properly referee all the papers at get written.  When fields rely solely on journals (eg, economics), backlogs can make it difficult for young researchers to get published —many have no publications when they finish their PhD. Also, accept rates of 5% force editors to be risk-averse.

The PC review system in theoretical CS is not perfect, but better than those in many other fields. The social pressure of a face-to-face PC meeting seems to make members take their reviewing quite seriously.

Also, many researchers seem happier to serve on a STOC/FOCS PC once every 3-4 years rather than on a journal board for 3-4 years. Perhaps humans prefer shorter but more intense pain to a longer and less intense one.  Or perhaps journal boards are less interesting because you end up handling papers in your own speciality, including those you already saw 2+ years ago.


(b)Incentive system for researchers to produce a substantial piece of work, and then write it up —sort of comprehensibly—in 10 pages.

The incentives in the arxiv model are quite the opposite —more frequent, insubstantial, and hastily written works.

The 10page limit —archaic relic of the papyrus era— and the PC model has led to our tradition of writing papers that are sort-of comprehensible to nonspecialists.


(c) Clearing point for deciding upon priority, novelty, correctness etc. of claimed results.

Conferences can do it faster and better than journals in most cases, at least under current rules (a jury of 20-25 PC members versus a jury of one editor and 2-3 random reviewers). The informal refereeing system at conferences at first glance seems to invite abuse but I can think of very few accepted papers at STOC/FOCS in the last 30 years that turned out to be very flawed (and often those were recognized as controversial when accepted).


(d)  A stamp of authority, or a recommendation if you will.

We increasingly need this guidance from conference PCs to stay afloat in the sea of new papers, especially outside our sub-specialities.  That is why I go to STOC/FOCS these days, not social networking (which is a nice bonus though). I could stay at home and watch videotaped talks but, really, who does that?

(e)  A synchronization mechanism for our field.


Is it just my imagination, or do conference deadlines actually enhance collaborations and improve productivity/creativity? Half-imagined results get fleshed out as people get together in the months or weeks before the deadline (and I am not referring to caffeine-fueled late-night finishes, which I avoid). We need this synchronization to structure our busy lives, and neither arxiv nor journals provide it. If you don’t care for the human weaknesses this argument stands on, I should mention Boaz Barak’s alternative explanation: sometimes correlated equilibria are superior to Nash equilibria.



Proposals to improve conferences in theoretical CS

The ongoing experimentation—a day of workshops, poster session, recorded talks, no paper proceedings, better feedback from PC—has been good.  Here are my thoughts for further improvement.


  • Keep the conference format (say 12 pages, 11pt).  But to reduce work and give an incentive to produce a readable version, make the submission format identical to the published format.(I admit to having done my share of grumbling about the conference format, but on balance it is important for our field that 50-page arxived papers should be accompanied by shorter, more readable, versions. If you think 12 pages is too few, try vying for the privilege of publishing your result in Science and Nature —in 2-4 pages!)


  • Increase number of acceptances moderately. (Beware though of Parkinson’s law: submissions increase to fill all available refereeing capacity. So don’t agonize if acceptance rates stay below 30% despite this increase.)But, reduce number of talks. (In other words, not all accepted papers are treated equally by the PC.) Have more plenary talks.


  • To combine some of the speed of arxiv with reliable timestamping, have two submission deadlines a year —papers appear on the website as soon as they are accepted in the first cycle, and papers rejected in the first cycle cannot be resubmitted for another year.  Variants of this model have been tested in other fields (databases, ML). This spreads out a PC’s work over a longer period, which has its pluses and minuses.
  • It would be nice —perhaps independent of conferences—to have a forum for posting reviews/comments on theory papers. (Hints of Le Cun’s ideas here.) To be useful we must avoid the vicious smallness of blog comments. Requiring posters to use verifiable identities should preclude the worst abuses (the system only needs to scale to a couple thousand users). 
  • Last but most important: keep the various points made in this article (or any other set of principles discussed and agreed upon collectively) in mind when proposing new changes.

Despite its small size, theoretical CS has been remarkably successful. An incredible edifice of ideas was created together with an open culture that values the need to address papers and talks to nonspecialists. This allows ideas and techniques to jump rapidly across subspecialities.  Theory conferences played an important role in creating that culture, and we should think hard about maintaining their good elements in the digital era.
To finish, I must admit that when I started this thought process (and discussions with colleagues) I started out somewhat skeptical  of conferences but ended up strongly in favor. I’m decided to be more willing to combat the cynicism I often see in such discussions; hence this blog post.

People’s views tend to be colored by their last conference rejection. Typically, senior people fume about youngsters who value technical sophistication or over conceptual contributions. Young researchers in turn feel anxious about being judged by a power structure that they don’t fully understand or feel part of. Such anxieties have existed since prehistoric times—there is no way to do research and not have it be misjudged at times. Cynicism is not a good response.

Further reading


“Time for computer science to grow up” by Lance Fortnow.

New publising model for computer science by Yann Le Cun

A full issue of Nature devoted to this topic:

(e.g., see

Some comic relief (article from 1967): The future of scientific journals. A computer-based system will enable a subscriber to receive a personalized stream of papers.

(Acknowledgements: Useful discussions in recent weeks with all my Princeton colleagues —Moses, Bernard, Avi, Mark, Zeev– and ex-Princetonians Boaz Barak and Ankur Moitra.)

40 thoughts on “Sanjeev Arora: Thoughts on Paper Publishing in the Digital Age

  1. I agree with many of the points in favor of conferences. I’ve proposed some of the changes you mention on previous STOC/FOCS PCs. In particular, I suggested accepting more papers by not assigning all accepted papers a talk. There was very significant push back. The main fear was that it would lead to an unfair two tier system. Authors would even go so far as to withdraw their papers if notified that they couldn’t give a talk. Indeed, it would almost certainly lead to some confusion initially. E.g., is a no-talk FOCS paper preferable to a talk at SODA?

    More generally, many people are worried that experimentation with the conference format hurts the perceived value of the conference. If in one year too many papers were accepted, so the reasoning goes, the signal of a STOC/FOCS paper would permanently be undermined.

    That’s why I believe lasting changes to the STOC/FOCS format will continue to be very slow. I’m more hopeful about external solutions catching on. I don’t think that they ultimately need to replace existing conferences. Authoritative reviewing forums could serve as a valuable aid to the already burdened PC. At the same time, I’m a bit worried about crowd solutions in general. There is no crowd. TCS is quite small. An even smaller number of us use Twitter or any given online platform. The number of Twitter mentions of a paper is essentially signal-free as of today. Similarly, online reviewing forums might well end up being very sparse with most papers not receiving a single review.

  2. I’m extremely uncomfortable with the idea of science being subject to blackhat SEO, astroturfing, online “management” (businesses like, and other market manipulation strategies commonly applied to the Web. Science doesn’t escape manipulation as it stands, but there’s no need to welcome in more damage.

  3. I would like to point out two advantages of our conference system.

    First, it has created an incentive for us to give good talks. I am often disappointed by presentations from those in communities without high-profile conference. We not only care more about our presentations: we learn how to speak well from watching our colleagues.

    Second, many of us, and especially me, learn much better from listening to a presentation than from reading a paper. Before attempting to read a new paper, I often look on-line for a talk about it. So, I think we should videotape all the talks.

  4. I agree with *a lot* of what Sanjeev has wrote, and mainly that the conference system (and FOCS/STOC in particular) served our community well and we should preserve its benefits. Naturally, I disagree with the recommendation about the submission format, and think that what should be incentivized is writing a good full version. I am not worried so much about the difficulty of producing 12 pages (this is done mechanically by most), but rather about its usefulness. I think that a 2-4 page summary could actually be much more useful, but this is what I expect the introduction to provide. In any case, this is a side issue to the post.

  5. Omer:

    You can incentivize full versions by requiring it. The 12 page version is for quick review by nonspecialists (kind of like how Science/Nature require full versions and then force you to send a brief version). Our field is now diverse enough that we need briefer versions of the paper. The intro is not enough.

    I recently had a great experience writing a brief version for a conference that brought a big sprawling manuscript under control. Made me a convert to the short format.

  6. Interesting post. However, I do not agree with most of what Sanjeev wrote. Note that he claims at the beginning that he is going to consider only the scientific merits of the conference system, but then he goes to list rather unscientific “advantages”, like “stamps of authority”, “recommendations”, etc. These cannot be justified scientifically, at least not directly.

    I also reject the political word “community” often used for TCS researchers. We are not a community. We are individual scientists who are doing research. The term “community” indicates indeed some power structures, which Sanjeev’s goes on to minimize their effect, if I understood well the post. In a community, there are people outside the community and established people inside, and new members are “accepted” to the community. This is not what is, or what should be, the case in science. In science everyone is welcome.

    Further, I reject the STOC-FOCS conference system as it is implemented nowadays: as a provincial ranking mechanism, dedicated mainly to American universities and relatives. Indeed, STOC, FOCS almost always take place in the US. Therefore, it cannot be considered a truly international event.

    I also reject the claim that competition and deadlines contribute to the scientific quality of works being done in TCS. Competition and strict deadlines might be adequate to engineering, but not to science, which, as Oded Goldreich puts it, has intellectual content, and not merely instrumental one. We can see a deterioration of scientific depth, originality and quality brought by the over competitive nature of STOC-FOCS TCS (see Goldreice’s essay about competition in TCS). As the late Mihai Patrascu once pointed, we can detect a crowed mentality in TCS, which is becoming more and more extreme as time goes by.

    Let me state that I support Lance Fortnow’s suggested model of turning back to journals. The benefit of journals is obvious and clear: a scientific contribution should be judged according to its own merit, and not relative to some artificial and inter-disciplinary comparisons. When we submit to a journal, our work is judged by its merit, and not based on what other papers are submitted. We have enough necessary competition in academia. We should strive to minimize any unnecessary competitions. Not increase it.

    1. Very nice comment, I fully agree with it! In particular, being European, I agree that the TCS community considered by Sanjeev seems to be more an US-based community. And I think it could be very interesting for everybody if there were more interactions between US and other continents. We sometimes hear about “American theory” versus “European theory”, and to my mind it is a bit a shame…

      I add that reading the post, I was wondering for many points “and what about maths?” (or another discipline, but I feel like maths are very close to TCS in terms of the way people individually do research). There are several points on the incentives it gives to people (to review, to write good papers, etc.): Is it an issue in maths that it is hard to find people to review papers? Or to write good papers? I do not think so, thus I think that Sanjeev’s arguments are arguments for solving problems that do not exist…

      1. Thanks Bruno for your support.

        For your argument about mathematics being successful without conferences: I believe that some people in TCS resent practices in mathematics. They view mathematics as a somewhat esoteric field that has no “influence”, on what they perceive wrongly (in my opinion) as the “real-world”. Where the “real world” for them is not much more than what other people (or the majority of people) care for. This leads quite obviously to a vicious circle. Since what the majority of people care for is also based on what the majority of people care for, and so forth. I believe this is among the reasons for the extreme emphasis on “staying relevant to the public”, “good talks for the large audience”, competitions and this sort of things.

        Of course, there are some advantages to this kind of approach, as demonstrated by Sanjeev’s and others in the comments above. But the question is whether there are better alternatives? To answer this we need to agree on the goals of the field themselves. But this is a matter of values. Some people really value success (measure by influence on other people; which leads to the vicious circle above) over any other internal measures to the field. I believe that this view is too radical.

        I support a middle ground where both influence (for practical reasons and as a way to regulate things) and internal justifications to TCS are valued. I believe that going for what Lance Fortnow suggested, i.e., the journal-oriented system, is one way to partially achieve this.

      2. I find your characterization of what some CS people think of math to be very strange. I never heard such comments from my TCS colleagues and I have a hard time understanding where this description of the criticism of math by TCS comes from. Perhaps this is the criticism of some more engineering parts of CS towards math (and for that matter towards TCS). If at all, the connections between TCS and math have dramatically strengthened in recent years and at times the border is very hard to define (with a flow of results and techniques in both directions).

      3. Omer, I’ve heard plenty of such criticism of anything that is “too abstract” or “too theoretical”, within TCS. I’m surprised you haven’t. Of course, it depends on who you’re talking to.

      4. I don’t doubt your account, but I have never heard such arguments and I have been exposed to a large number of members of the FOCS/STOC community. To the point, this criticism makes no sense when it comes from TCS. It is clear to me that the two of us have vastly different experiences, I would love to hear some of your research-life stories, and share them with others through our blog project.

  7. I really like the idea of having two submission deadlines over a year for a single conference! You might then have a paper and its follow-up(s) appearing in the same conference, which would be a fun thing to have happen. But then, it should be a requirement that once the accepted papers list is out for each of the deadlines, the authors should put up a full version online.

  8. As a “young researcher” who is “being judged by a power structure that I don’t fully understand or feel part of”, I don’t think I have enough experience to comment on how our conference and journal publication system could be improved. However, I am very excited about the future digital tools that will complement the existing publication system.

    First, I can’t imagine what it would be like without arXiv to publish papers and Google to find papers. In addition to these modern wonders, I think it is only a matter of time until there is going to be place online to publicly share paper reviews. I would love to contribute to this.

    About “(a) Incentive system for good researchers to (sort-of) review lots of papers.” I think a website (like arXiv) that “kept score” (like StackExchange or the seed ratio in a BitTorrent swam) might work well here. To this website, researchers submit both their own research papers and reviews of other research papers. However, you are not allowed to submit a research paper of your own if your ratio of “published papers” to “reviewed papers” is too low.

    Of course, one critique of this idea is that it seems likely that people will contribute low-quality reviews in order to be able to publish their own work. However, finding the right incentives to elicit high-quality reviews applies equally well to any system that accepts unsolicited reviews (not just my idea above).

    The other critique that comes to mind is that young researchers will have a hard time getting into this community. I have two counter-points to this. First, (like StackExchange), one can easily tweak the constants that define what actions you are allowed to make with a new account (thought I don’t claim to know what those constants should be). Second, I think young researchers are perfectly capable of contributing high-quality reviews, even before they have a paper of their own (which will allow them more freedom to publish their own work on such a site).

    In the current publication system, reviewers are (initially) selected based on who the PC/editor thinks would be able to provide a good review. Of course, young researchers are often left out of this process because they are neither well-known nor have proven to the PC/editor that they would be able to provide a good review. In contrast, if young researchers were allowed to review whatever paper they choose, then they should be able to find some paper for which they are capable of providing a good review. For example, most of my papers are extensions of previous work. By the time we were ready to publish (the arXiv version) of these papers, I could have easily written high-quality reviews of the one or two papers that we extended.

  9. To CS Prof: Thanks for the correction. “Theoretical CS” in this post is interpreted as “US Theory” (also popular in Israel and pockets of Europe and Asia) or as Moshe Vardi calls it, “Theory A”. Also the same “theory” that this blog’s name refers to. Wasn’t aware that any theory B readers would follow this blog.

    Tyson: You would be surprised how many of the reviewers are young (say < 5 years past their PhD). I don't know too many other fields where young researchers hold so much sway over the field. Just look around the room at STOC/FOCS/SODA. People over 40 are a definite minority.

    There's nothing wrong with web reviews. My point is that the process will just be uneven and even less fair than today. If you are an unknown researcher, good luck with getting good three reasonable-quality reviews of your work (which is what the good conference manages to deliver most of the time).

    1. Hello Sanjeev. It was not a correction. I am part of “theory A” myself. It was a criticism of STOC-FOCS as not a truly international system for all scientists in TCS, even within “theory A”. I assume that the commenter Bruno is also in “theory A”. STOC-FOCS is a system for a specific “community” centered in the US. In any case, this was just one part of my rejection of the current conference system.

  10. If we are rethinking the publication system, why not take one step back? Publishing papers is just a means to achieving the ends of increase of knowledge. We don’t have to confine ourselves to these specific means, why not explore non-paper alternatives?

    For instance, Q&As on cstheory look nothing like papers, and yet they expand our understanding of theoretical computer science and even being incorporated into the existing paradigm through citations. Why not promote a system that recognizes these sort of activities not as time wasted, but as the useful contributions to community that they are? This produces a completely different, and much more focused, system for disseminating knowledge in TCS.

    Similar thing for blogs, for instance, you wrote this article in a blog because it is a better way to reach people and get instant feedback than sending it to a journal or other resources that we would have turned to before the internet. Similar can be done for research, producing not only open publishing, but open knowledge generation. The best examples of this are the polymath projects it the math community, The comments threads and wikis set up for that generate new knowledge, and they are afterwards published in math journals only as a ceremony and a way to keen non-bloggers in the loop. Why not encourage similar things in TCS?

    I would be very interested to see popular blogs acting as a sort-of publication venue with many guest-posters contributing interesting questions, partial results, post-publication review, or mini-surveys. Of course, moderation would have to be stepped up in the comments to avoid some of the nasty anonymous stuff that happens, but that isn’t hard to implement.

  11. I agree with most of Sanjeev’s points, and especially with the many positive aspects of conferences both as a mechanism to highlight papers and an event where researchers get together and learn about new results. I also share with Dan the impression that: (a) talks are often better than papers to convey at least the high level information and (b) CS talks (and introductions to papers) seem on average better than those of other fields that are more journal based.

    One point re “Theory A” vs “Theory B”, I must admit I don’t always know where those boundaries are (e.g., is the work on CSP dichotomy I blogged about here and some of which appeared in STOC/FOCS Theory A or Theory B?), but we should strive to be inclusive and not set up barriers in our conferences and in this blog.

    My feeling is that some of the criticisms leveled against conferences of competitiveness and erroneous or subjective reviews are inherent to any system of allocating the scarce resource of attention. People have the same amount of time to read and review papers, regardless of whether they do it through conferences, journals, archives, or new web-based mechanisms, and as far as I know, such issues arise in all the scientific fields.

    1. The increase in competitiveness of STOC/FOCS/SODA is an unavoidable consequence of the community getting bigger. I don’t think major changes need to be done though, after all, we established a new conference (ITCS) so more papers are being published, and we also see a steady rise in the quality of the ‘second tier’ conferences, which is a good thing.
      I mainly want to echo previous comments that the way papers are published is a part of a larger and delicate echo-system. Changing it would surely have unintended (and unforseen) consequences, on the quality of presentations, the way researchers collaborate, the way researchers are measured in the job market and more. When comparing to other fields, my general impression is that TCS is doing pretty good. We should definitely strive to have a better system, but I would support slow tweaking over a revolution.

      1. The increase in competitiveness of STOC/FOCS/SODA is an unavoidable consequence of the community getting bigger.

        Unavoidable? All we need to do is accept more papers and presto! we avoided it.

        It is important to note that STOC grew steadily in number of papers from its first edition in 1969 (23 papers) until 1994 (81 papers) at which point growth stopped arbitrarily. There is no reason to believe that keeping the size of the conference constant since then in the presence of growth is in any way preferable to what was done before.

        To the contrary, STOC’s role as a social networking venue (one of the main objectives of a conference, as pointed out by Sanjeev) has diminished due to its lack of growth. As others have observed elsewhere, STOC used to be a must-go conference that everyone would attend, paper or no paper. This is no longer the case today.

  12. My feeling is that some of the criticisms leveled against conferences of competitiveness and erroneous or subjective reviews are inherent to any system of allocating the scarce resource of attention.

    Indeed, however you have to ask “does being extremely selective increase or decrease the effects of an erroneous review?”

    At the top conferences, over the last fifteen years, we have effectively moved from a majority vote (which is resilient to small errors) to a unanimous vote, which it isn’t. This is a bad effect of the top conferences which is not related to scarce attention, and one which could easily be solved.

    We could for example, increase the number of acceptances in STOC and FOCS in a minor way, say 10% more papers for each of the next three years, thus reducing the number of incorrect rejections and the overload on said reviewers, since we wouldn’t have to re-referee the paper for another venue.

  13. CS Prof. and Alex, Some replies:

    Let me start with the agreement. I think that over competitiveness is bad for individual researchers and to a field of research. I also agree with Alex that accepting more papers to FOCS/STOC would be good. In fact I was shooting for more papers in FOCS 2013, but failed. This was not the result of some preset number or of direct competition between papers. We discussed and decided every paper individually, and I guess our internalized bar was too high. I take the blame. By the way, acceptance did not require unanimous decision and many times one enthusiastic PC member made all the difference.

    I want to mention that communities focused on journals are many times as competitive or even more so. There are communities where a career is made or broken on a single/few publications in journal x (where x is Econometrica/Annals/Science/…). In addition, journals are controlled by fewer people that are usually more senior and seat on the editorial boards for many, many years. Our conference system distribute the decisions much more and let more junior researchers make a large part of the decisions. (By the way, if anything younger researchers are more critical than more senior researchers.)

    Finally, the word community is one of my favorites. I think it is sad that your first association with this word is of power structures and exclusiveness. I can think of many communities that are missionary rather than exclusive. Like it or not, research is not a communal activity, and research in a vibrant young field like CS is even more so. We are individual researchers but in addition members of various research communities focused around fields of research or around publication venues. The all purpose of FOCS/STOC is helping TCS keep as a community rather than split into its sub-communities as there is a huge scientific benefit from the flow of ideas between these sub-communities. FOCS/STOC is in some cases doing a better job in it and in other cases a worse job but this is its purpose. So perhaps the power structure should be weaker and more distributed perhaps the community should be more exclusive but it is a community and we should be fighting for its success.

    1. By the way, acceptance did not require unanimous decision and many times one enthusiastic PC member made all the difference.

      While I do not question that this occurs in some instances as I myself have lobbied papers away from the reject column all the way to the best paper award slot it is far from the norm: the majority of papers with a single bad review simply die a quiet dead.

      Certainly my experience as PC member in numerous conferences at all levels is that as we move up the acceptance threshold moves roughly from a min easychair accept score of 1 (small/local conference or specialized workshop) to 1.5 (international well known conference) to 2.x (top conferences in the field). The last score translates pretty much to an accept across the board from all referees.

      So all that is needed for the paper to be rejected is a single reviewer who misunderstood some point, with the two other positive but not overly enthusiastic reviewers lacking the time to double check and counterweigh those criticisms.

      1. I disagree with this description. In our PC, and in many others, controversial papers (papers with disagreement) got most of our attention and there was nothing automatic about their acceptance/rejection. It was also far from our norm to outsource our decisions to external reviewers or to give anyone a veto power. Surrendering to the enthusiastic PC member was much more the norm than surrendering to the negative PC member. We were very far from perfect and may have made many mistakes, but we invested months of hard work in a very deliberate and thoughtful attempt to be fair and thorough. I am not sure that any other system would have done better. I agree that accepting more papers would have been good, but we are not so far from a natural limit imposed by 3 days and two sessions. Perhaps we should go with 3 sessions, let some papers be only presented in posters or any other solution. But all of those come with a price which so far PCs have been reluctant to pay. Indeed, such changes are a matter for the community at large (rather than for an individual PC) and this is the reason for this post (before the FOCS business meeting).

      2. By bad review I’m not talking of a paper with scores 2 (correctly given), 2 (correctly given), and -2 (incorrectly given). This surely gets a lot of attention as you say.

        What I’m talking about is a paper with scores 2 (correct), 1 (tough but fair), and 1 (incorrect). Nothing stands much in this case and paper is likely rejected, while if reviewer 3 hadn’t misunderstood something in his/her haste reviews would have been 2 (correct), 1 (tough but fair) and 3 (correct) and paper likely gets in.

        In other words, very good/very bad papers are mostly properly dealt with. It’s the ones in the middle where the errors are amplified and it essentially becomes a random process.

  14. Alex, there is no way to avoid errors. I am not even sure they should be called “errors” in most cases. In any judging system that rejects a majority of submissions, a significant number of papers will end up at the borderline, and some will be rejected.

    I agree with you that we should increase the number of accepts a bit —20 or 25. This may involve adding a third parallel session or a poster session or something. But don’t be under any illusions that it would mitigate the above phenomenon.

    1. Alex, there is no way to avoid errors.

      I don’t think anyone here ever asked for a perfect system, we are all seeking a better system an we are looking at proposals such as yours.

      In the case of decisions, surely we all here have dealt at one time or another with probabilistic algorithms and the classical ways to amplify their probabilities of being correct. Why are we not applying those here?

      In fact, much to the contrary we seem to purposely choose mechanisms that amplify the probability of error, rather than the probability of correctness.

      Stating that “there is no perfect system” or “we only make mistakes on borderline papers” are post-facto rationalizations to justify a situation into which we have devolved as a consequence of the no-growth policy for STOC instituted in 1994.

      1. But the hypothetical example you mention will arise in every reviewing system, no matter how perfect. So how is it a proof of the current system being wrong? (Note that I am agreeing with your point that we should increase the # of accepts; just disagreeing that this will solve the problem.)

        I agree that somebody should look at the problem of accurate paper ranking given imperfect reviews. I wouldn’t be surprised if some ML person has already done that (make a database of reviewers and their past rankings and their accuracy, etc., and solve the ML problem determinining a more accurate rating). There’s a paper idea for somebody….

      2. Am I missing something? It seems to me that simply moving from a “unanimous accept” to a “majority accept” score reduces the probability of error tremendously. Let’s say 1 in 10 reviews is incorrect (shadow PC exercises suggest the number is much higher). Unanimity means fully 25% (~0.9^3) of good papers are rejected. Majority means less than 3% of good papers are rejected.

        In that vein, I recently overheard a conversation about an equally selective conference, one of the SIG____ conferences that shall remain nameless to protect the guilty. The authors were optimizing the submission under the knowledge that person X would very likely be a reviewer and X is well known to like a specific type of explanation. The authors also knew that a single bad review, even from an idiosyncratic one, is enough to kill the paper there and they had to tailor their submissions to X’s tastes. To me this is highly undesirable from a scientific perspective and, sadly, the direction in which STOC and FOCS will go if we don’t take steps to reduce the influence of a single bad review.

      3. I think you misunderstand. Omer’s post suggests that for borderline papers it suffices to have *one* enthusiastic review to get in. Signing off of this thread now for a few days.

      4. Indeed, I do not agree with Alex’s description of the selection process in FOCS/STOC, and it is certainly much less automatic than Alex seems to suggest. I still agree with his conclusion that a modest increase in the number of papers accepted is a good thing. Some concrete comments:

        (1) papers with disagreement reach a discussion in the PC meeting, they are never decided according to some function of the 3 assigned PC members scores (and so there is no requirement of unanimous accept and no way to implement a majority) (2) Once in the PC meeting the paper is discussed and the PC members try to convince each other. PC members ask questions, make comments and at time the PC postpones the discussion to get some additional information. If consensus is not reached then eventually there is a vote by the entire PC. The dynamics changes from one PC to the other, but it usually does not reflect the unanimous accept assumption. (3) If a PC members strongly objects and the rest are lukewarm then the paper will probably be rejected. This is fine – in STOC/FOCS someone needs to care that the paper will be accepted. But even solid support (rather than enthusiasm) can convince the committee. (4) The selection is not perfect. The main thing is that too many papers need to be discussed and not all of them get the same level of scrutiny. Another problem is that some PC members are more eloquent and opinionated than others, this can make a difference. There are other problems, and most would not be solved by moderate increase in acceptance numbers. (5) There isn’t really a ground truth, but if there was then accepting more papers would most likely reduce the number of papers that are “wrongly rejected.” Given that the submission pool is so strong, accepting more papers does not need to weaken the quality of the conference (at least not by much). (6) Even more important – accepting more papers will allow more people to be active participants in FOCS/STOC, which is very important in my mind.

      5. [Several comments describing STOC/FOCS deleted]

        It that is how they operate, that is different from all other program committees I’ve served on. In my experience papers right on the boundary get a lot of attention; papers with vastly differing scores do so as well (but often only after prodding from the chair, as I’ve learned first hand as a chair myself); and lastly papers in the easychair scale a point or more away from the accept/reject boundary (in either direction) are often glossed over and semi-automatically accepted/rejected.

        in STOC/FOCS someone needs to care that the paper will be accepted.

        If this is indeed the case, then it seems to me that any area who is not well represented in the program committee will lack such such champions.

        In other words, if we are to stand by the “paper needs a champion to be accepted” criterion then we need a larger program committee, don’t you think?

        p.s. I don’t intend any of the comments above to be specific to FOCS’13 or the job done by Omer, which as far as I can tell was excellent. I’m more interested on what STOC/FOCS has (d)evolved into over the years and how to make sure that they keep growing–literally and figuratively speaking.

      6. Thanks for the compliment. I did not take your criticism personally and I think it is all very legitimate concerns. I don’t think my PC was so unique. I believe that there are a lot of stereotypes about FOCS/STOC that are unjustified and not useful. Indeed, papers whose topic is not covered by the PC are a problem, as in that case the reliance on external reviewers is larger. But it does not mean that they do not have champions. I deliberately wrote “care about a paper” rather than “enthusiastic”. The percentage of not covered papers is not high and PC chairs do try to respond to changing trends. But I agree that this is somewhat of a problem. Increasing the PC have both advantages and disadvantages, and I heard people argue passionately in both directions.

  15. I agree with most of the points raised here about the value of our current conference system.

    However, I do worry about increasing the number of acceptances as a way of coping with the strains brought on by the growth of the field. More papers and more parallel sessions can diminish the role that conferences play in bringing the different subspecialities together and in drawing attention to papers outside their specialty. Instead, I am attracted to the idea of a “federated theory conference,” with numerous specialized conferences being co-located before and after FOCS/STOC, as an attractive way to accomodate a larger community and larger research output while maintaining the unique role that FOCS/STOC play. More details at

    I also very much agree that we need to incentivize the writing and dissemination of full versions of papers, ideally submitted to journals for a careful review. Like Sanjeev,
    my tendency is to move towards requiring that full versions be available for all conference papers (see for more discussion of this issue), while not eliminating the 10-12 page digest we currently have (which I also find to be very valuable for non-specialist reviewers and readers).

  16. I agree with most of the points raised here about the value of our current conference system.

    However, I do worry about increasing the number of acceptances as a way of coping with the strains brought on by the growth of the field. More papers and more parallel sessions can diminish the role that conferences play in bringing the different subspecialities together and in drawing attention to papers outside their specialty. Instead, I am attracted to the idea of a “federated theory conference,” with numerous specialized conferences being co-located before and after FOCS/STOC, as an attractive way to accomodate a larger community and larger research output while maintaining the unique role that FOCS/STOC play. More details at

    I also very much agree that we need to incentivize the writing and dissemination of full versions of papers, ideally submitted to journals for a careful review. Like Sanjeev,
    my tendency is to move towards requiring that full versions be available for all conference papers (see for more discussion of this issue), while not eliminating the 10-12 page digest we currently have (which I also find to be very valuable for non-specialist reviewers and readers, though it’s good to experiment with alternative approaches as done in FOCS `13).

  17. Clarification on page limit idea.

    Had some clarifying discussions with people (Omer and James) about page limit, both of them opposed. So here’s my rationale.

    Their rationale (to paraphrase) was: people know best how to present their own work. Leave things to them.

    Wrong on several counts.

    (a) “People know best.”

    The following ritual occurs every year. Princeton/IAS postdocs (all of whom I respect a lot) show me their research statement which they plan to use in job applications. Usually it is 7 pages. I tell them: “Make it 2-3 pages or its useless.” They are horrified. After several rounds of revisions they agree it is became better and crisper.

    There’s a reason your 7th grade English teacher —at least if you had a good one—redinked yours meandering writings: briefer is better.

    (b) “System should incentivize authors to write full versions”

    Absolutely. But put the full version on arxiv for experts and posterity. But, also prepare a briefer version. I guarantee that the process of writing a shorter version is going to give you new insights into how the result works, and also how to present it in your talks. It is not time wasted.

    (c) Finally, shorter versions help us stay sane in the arxiv age when we are awash in pages and pages of latex.

    Shorter papers are also more accessible to nonexperts. At least that’s what I find when I try to read papers in other fields (outside theory).

    1. Hi Sanjeev,

      Your paraphrasing does not reflect my opinion at the least so let me clarify. I will be somewhat brief as I already devoted a couple of posts to this matter. While I agree with much of your overall goals, I completely disagree about the usefulness of the page limit as a way of reaching these goals. But this is not a theoretical argument, we already have a lot of evidence. My impression of the page limit rules used for submissions in our community is that it was a colossal failure, at least in the last 15 years. It did not help create concise and accessible presentations but only created hard to review papers. The reason is simple: the vast majority (essentially all) of the submission versions were created in a completely automatic manner by methods that only hurt readability. It is nice and idealistic to hope for something else, but I see no reason to expect methods that have failed for decades to somehow succeed now. I think a new approach is needed, and this is what I am trying to promote. The evidence from FOCS 2013 is that things have already improved (and on average reviewing seemed to have been easier), but this is a new policy and I think further improvement will be made if we stick to it for a while (and I am not asking for 15 years).

      The signal that needs to be sent to the community is that presentation is important. This means that we should reject papers that are badly written and we should do it more aggressively, constantly increasing our expectations. Indeed, the no page limit is a twofold change – removing technical requirements but increasing fundamental expectations. As our call for papers indicated (and this is phrased in stronger terms for STOC 2013) part of the requirements are on the introduction which should be the concise and generally accessible of the paper (though other parts should be accessible too and include lower level intuitions). Have a look at how it is paraphrased …

      Finally, I do not think that it is enough to incentivize the existence of full versions on archives as many of those are below minimal standards (causing real damage to the research in some subareas). Instead, I would like to incentivize well written full versions and I think this is a nice side benefit of our approach (compared to the sloppy and incomplete writing the page limit rule has incentivized). Nevertheless, this is not my main motivation.

      1. OK, we are in great agreement about the goals. I just know from long experience that most people benefit from a page limit, and they have been getting mixed signals from the PC in recent years. It is generally assumed that the page limit is a useless relic from the papyrus era. It is more and more common knowledge that the PCs are willing to look at 50-page papers. But page limits have a timeless purpose that I am only now beginning to understand. And it is about more than just writing a good intro.

        Anyway, this is perhaps not the biggest or most important aspect of rethinking the conferences.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s