Tips for future FOCS/STOC program chairs

There are only two FOCS/STOC chairs per year, and most would do just fine without my unsolicited advice. Nevertheless, I thought it might make sense to write down some of my thoughts after chairing FOCS 2014, and perhaps even some people that are not program chairs will find it of interest.

This is based on advice I received from many people, some of which are mentioned by name but most aren’t. However the opinions here are my own. Others’ opinions (including those of the people mentioned) may vary: there is more than one way to run this process. Despite the length of this document, there are still many points and details that I omitted. If you are going to be a program chair, feel free to contact me for more advice, examples of forms, documents, and my (badly written and comment-free) code for scripts and website. You can also ask questions in the comments below.

Many discussions on conferences focus on issues of process – one-tier vs two-tier PC, anonymous submissions, page limits, conflicts of interest rules etc… These are all important issues and as you’ll see below I do have my own opinions on them. But I believe that to a first approximation the only thing that matters is the work that is put in collectively by the chair, PC, and reviewers. If the papers get reviewed by the right people that spend sufficient time to do a quality job then the results will probably be fine no matter what process you use; otherwise no process will save you. To a large degree, whether this happens is controlled by the amount of effort you put in as chair in first selecting the right people and then constantly staying on top of the process making sure that no paper falls between the cracks.

There are two common arguments why things will turn out fine even if you don’t work as hard as you could. I disagree with them both:

“No matter what, the good papers will get in, the bad papers will be rejected, and the only job of the PC is to figure out the decisions for the papers in the middle.”

This is patently false as we all know famous examples of great papers that were nonetheless rejected from conferences. There is no reason to think that such colossal mistakes were unavoidable and could not have been prevented by the PC working harder. For every one of those famous examples, there are many other “near misses” where the PC made the right decision only because of the diligence and persistence of one person.

“Even if you reject a great paper, it’s not so bad, since the author can always submit it to the next conference.”

Sure the author would do fine, but this is not the point. You don’t work for the authors but for the community and it’s not the conference that honors the paper – it is the paper honors the conference. So when a great paper is omitted from the program it is the conference, and by extension the community, that suffers. This is true even if the paper appears in the next conference but there are also other cases: the paper might be rejected again, or (increasingly likely these days) the author might submit the paper to a specialized conference, and so it might take much more time for the work to get known by the general theoretical CS community.

With general ruminations out of the way, here are some concrete tips in chronological order, from the time you are asked to be a program chair to the actual conference.

Step 0: Get invited early

Ideally, you ought to know you’ll be chair sufficient time in advance so you can slowly and deliberately assemble your committee. People are also more likely to commit to the PC when you ask them far in advance (if only because it’s much harder to think of excuses to refuse that will apply so far into the future..). Of course, you don’t control when you are asked to become PC chair, but I hope that future SIGACT/IEEE TCMF chairs will keep this in mind.

Step 1: Set the PC meeting date

I think it’s important to get 100% attendance at the physical PC meeting, and so you want to know the date before you invite people to join the PC. I had about 11.5 weeks between the submission deadline and the meeting which was a good amount of time, though it would perhaps have been better (if possible) to get an extra week. Run potential dates with knowledgeable people to make sure they don’t conflict with any other conferences or activities that some of your potential PC members might be attending. (For example, Paul Beame saved me from scheduling my meeting at the same time as COLT.) My PC meeting was two days –9am-7pm Saturday and 9am-5pm Sunday (so people can catch flights back) but in retrospect we were a little tight on time, and I should have added a working dinner on Friday night as well. Once you set the date you can already book the conference room and (optionally) reserve a room block at a hotel.

Note that I’m assuming here you are using a traditional single-tier PC. I was an executive committee member of STOC 2013 that used a two-tier PC and I think the end result was fine, though I do see an important advantage for a physical meeting in ensuring that the program is more cohesive. There is something to maintaining the invariant that for every paper eventually included in the program, someone had to stand up and make the case to 20+ people that it is important that this result is broadcasted to the TCS community. More importantly, I believe that very few people in our community have the management skills to effectively run a two tier organization of 60+ PC members. Luckily, Joan Feigenbaum (the STOC 2013 chair) is one of those people.

Step 2: Choose your PC

This may be the most important choice you make, and will determine much of how well the rest of the process goes. You want people that are not only great scientists, but also have good judgment, ethics, and are nice and responsible people that answer their emails promptly and you wouldn’t mind being cooped up with them in a room for 20+ hours.

I first wanted to get a large pool of potential names. I started by looking at every paper in the last two STOC/FOCS’s, thinking who would have been an ideal person to review that paper (for the purposes of this exercise, that ideal person can also be the author). I also added many more names by thinking of various research areas, consulting an (unfortunately very partial) list David Johnson maintains of authors who didn’t serve on PC’s and also getting some suggestions from some of the first few people I invited to my PC. You can also look at people who served on PC’s of other theory conferences. I ended up with a list of about 80 names, out of which I invited a little under 40 to end up with a PC of 22 people (not including myself), which I think was a good size. (For STOC, that typically has more submissions than FOCS, you might want to have a couple more people though you still want to keep the size manageable.) It can be nice to have a person that served in the PC of the previous conference (or at least you can use this as an excuse to get a great person to agree despite them having served in the last conference..)

The invitations were an iterative progress, where I started with a few people and based on their answers continued inviting others. There are many factors to think about- mixture of areas, institutions, gender and other diversity etc.. (For example, one form of diversity that, as pointed out by Sanjeev Arora, I neglected to consider explicitly was age –I had very few people over 45, mostly because more senior people were much more likely to decline invitations.) To make sure I covered all areas, I made a spreadsheet with a row for every PC member I was seriously considering and a column for every sub-area of theoretical CS. In the entry corresponding to the X-th row and Y-th column I wrote my (pulled out of a hat) estimate for the fraction of papers in the area Y that X would be able to review. I used statistics on submissions from previous conferences to get an estimation of the expected number of papers in each area Y, and so this matrix was useful for me in ensuring that I have enough PC members to review papers in each topic. (As an aside, please do collect information on submission topics, as it will be useful for future chairs.)

Some tips for inviting members:

1. Aim high – don’t hesitate to ask your dream PC members, even if they are clearly so overworked that you think there’s no chance they’ll agree. Also, it doesn’t hurt to beg J
2. Ask around – both to get names of people that you don’t know personally and to verify that people you are considering are in fact responsible and conscientious. Ask PC chairs of past conferences for both recommendations of great people and to check out how someone you are considering functioned. The system is not really resilient to a PC member that completely slacks off in his or her assignment, so you want to make sure you don’t have such people on your PC. (Thankfully, I didn’t.)

While most of the PC should be people from across many areas and institutions, it’s good to have at least one or two close colleagues that you know and trust so you’ll always have someone to consult with.

Once you have the PC, you’d want to setup a mailing list to communicate with them and perhaps also a shared calendar. It makes sense to fix in the beginning all dates – not just the deadline and PC meeting but also the deadlines for “shell reviews” (see more below), reviews from external consultants, deadline to post internal reviews, and more. One lesson I learned from Joan Feigenbaum- never give a deadline without a precise time (e.g., the deadline for reviews should not be, say, June 1^st, but rather June 1^st 8pm Eastern Time). When you’re setting these deadlines, make sure that they are set early enough so that you can still recover if some people fail to meet them. Rather than (or, more accurately, in addition to) sending my PC members lots of individual emails, I prepared a single document containing all important dates, policies, and instructions. My deadlines were as follows (D+x means x days after deadline):

• D+2: Submit reviewing preferences
• D+19: Post “shell reviews” for all papers
• D+33: Deadline for external reviewers
• D+45: Deadline to post all reviews, electronic discussion begins
• D+80: PC meeting
• D+90: Notification (though I ended up sending notifications 3 days after the meeting)

 

Step 3: Prepare the call for papers

Once you have a PC, you can start discussing with them the call for papers. Much can be copied from previous conferences, but there are some decisions that need to be made. Following Omer Reingold’s lead, we decided to have no page limit for submissions. I think it was the right decision. While it might seem like shorter papers are easier to review, when a page limit is imposed what ends up happening with many submissions is that a few hours before the deadline the authors cut and paste sections into the appendix. In my experience this copy-paste transformation almost always makes the paper harder to read rather than easier. After all, reviewers are never obligated to read the whole paper and can always skip sections on their own. Given that we skipped some of those formal rules, I felt comfortable insisting that authors read my blog post with suggestions on writing their submissions. In retrospect, I should have included in it some non-binding concrete suggestions. For example suggesting that by page 5-6 the paper should be done with the introduction so that a non-expert should know what they did (i.e., statement of main result), and why it is novel and interesting, and that in a typical paper by page 12-15 or so there should be enough information about the proof for an expert to be convinced that it is most likely correct.

There’s a school of thought that the submission should have the same format as the proceeding version, and some people even believe that it should not contain appendices. I disagree. It’s true that the final “camera-ready” short version (or the conference talk for that matter) often omits many details of the proof, but I think the maxim “trust but verify” holds true here as well. I can sit back in the conference talk and trust the author that the omitted details are immaterial knowing that the reviewers had all those details at their disposal. While we don’t check proofs line by line, for every FOCS paper we had expert reviewers who believed that the proof is fundamentally correct, and I don’t know how to reach that state without giving those reviewers the full paper. Besides, in my experience more often than not the full version of a paper is a more enjoyable than the compressed proceeding version (especially with its ugly two column format).

There are some other issues you may want to mention in the call for papers. I thought of making a requirement that people post accepted papers on the arXiv but many people on my PC objected on the grounds that this would be unenforceable. I still think that if we required every author to check a box upon submission promising to do so, then the vast majority of those authors would have fulfilled this promise.

Salil Vadhan once suggested requiring authors to post their papers on the arXiv before submission. I actually think this idea makes a lot of sense, and makes formal what should be the right workflow- you write a paper, post it online, and then submit it to the next deadline. It also avoids the problem where a paper is rejected multiple times but never posted online, and so it is in a sort of “limbo state” where many people know of its existence, but no one can cite it or use it for follow-up work. But regardless of Salil’s proposal’s merits, this is probably not a decision that a chair or a PC should make on their own.

Following discussion in the FOCS 2013 business meeting, I added language to the call for papers that we might consider submissions that appeared previously in a journal that has a very different format and audience (such as Science or Nature). It is important to emphasize that authors should notify the PC regarding any related publications or submissions to conferences or journals that they are involved in or otherwise are aware of.

At this point you also need to setup the website for the conference. Since you control most of the content in the website, and you’ll be the one most frequently updating it, I believe it’s best if you own the website, while including in it links to sub-pages managed by the local chairs. The wonderful Wolfgang Richter gave me access to the codebase for the SOSP ’13 website, and I wrote some scripts of my own (e.g., to auto generate the program and list of accepted papers from the hotcrp output) that I’d be happy to share. The website is mobile-friendly and I also had a mobile app for the conference (more on that later).

Step 4: Make some policy decisions, plan review form, etc..

The weeks before the deadline are a good time to have some discussions about policy issues. Some policy discussions are best made with particular examples, but others are better made in the abstract when it’s still possible to separate the principles from the identity of the paper (and the authors). In particular, you’d want to settle the conflict of interest rules ahead of the deadline. There are two types of potential conflicts. A standard conflict is where a person might have some bias one way or another, but there is no fear that they would leak the discussion to the author. In a small community such as ours, such biases are very common. A strong conflict is when a person is so close to the author (e.g. is a close family member such as a spouse, child/parent or sibling, is a current/recent student/advisor, or an extremely close personal friend) that other PC members would not be comfortable discussing the paper honestly in their presence.

I think the model that works best for theory conference is to use a disclosure based mechanism to handle standard conflicts, complementing the exclusion based mechanism that one must of course use for strong conflicts. In particular, following some other conferences’ examples, every review in FOCS 2014 needed to include a “conflict of interest statement” disclosing any biases between the reviewer and the authors or the particular paper. (For example, one common potentially negative bias is if the reviewer is the author of a different submission on a similar topic.) Such disclosures allow the program committee to put the review in context, while not giving up on much needed expertise. Indeed, for many theory papers, there are a handful or so of top experts in that particular area, whose perspective is extremely useful, even if one needs to take into account some biases including people’s natural tendency to like their own area. In addition, using strong conflicts sparingly helps ensuring that most PC members have a global view of the program and pool of papers. That said, a PC member always had the option of declaring a strong conflict if they prefer not to have any knowledge of discussions on a paper. (In particular, in our conference we used this for faculty in the same department, though not for example if a PC member was a faculty member and the author was a student or a postdoc in the same department but advised by someone else. If you are going to vote on someone tenure’s case, or they on yours, you might feel more comfortable not participating in a discussion on their paper.)

You also need to choose the conference software. Two choices that people seem happy with are Shai Halevi’s system and Eddie Kohler’s hotcrp. Based on recommendations from some of my PC members, I used the latter system and was very happy with it. One nice feature is that it allows extraction of much of its data in a comma-separated file format. (As far as I remember I only needed twice to write mini scripts to extract data from the HTML code.) We also used its tagging features very extensively.  Moses Charikar volunteered to host it at Princeton University and staff member Joseph Crouthamel was a great help. (I initially toyed with the idea of setting the deadline time for the convenience of the university in the world that can organize the largest group for a post-deadline beer outing, but eventually set the deadline for a time that would be early enough so that Joe was still in the office.)

Use the time before the deadline to test the system and set it up.  I tried to keep the reviewing form simple without too many fields (just score, confidence, visible review and PC-only comments). I copied the score system used by Salil Vadhan in STOC 2011 since I liked its explicit descriptions of the meaning of each grade (I toyed with the idea of removing numerical scores altogether, though ended up keeping them and I think they did serve some use in searching and sorting at some stages of the process). In particular if you’re using hotcrp, make sure that you understand its various options for review and comment visibility for PC members and authors which might be a bit counter-intuitive for theory conference. Regardless of the software you choose, you might find it helpful at some point to write some simple code to augment it. If you are not comfortable with this, you may want to make sure you have access to someone who is. Or you can get more comfortable: I used chairing as an opportunity to learn a bit of Python. That said, there are PC chairs that managed just fine without writing any scripts and only using the ready-made software which is in fact quite good.

There are some other ideas you might consider. At FOCS 2009 we asked authors to provide in addition to the submission a 2-page summary of their paper. I thought they were nice to have but wasn’t sure their utility was worth the effort of the authors. Another idea I heard which is possibly more appealing is for authors to provide a slide deck (perhaps with recorded narration). I can imagine that quickly going through this deck before starting to review the paper can help a lot in quickly getting a global sense of what’s it about.

You also need to keep in constant touch with the local organizers, make sure that the arrangements with hotel etc.. are proceeding as planned, and that the contract with the publisher is signed on time.

 

Step 5: Reviewer assignment

I gave PC members 48 hours to indicate their preferences, and then gave myself 2.5 days to assign the review. (In addition to these, before the deadline I asked PC members to send me 1-2 sentences describing their areas of expertise.) I made the assignments manually. It took me about 12 hours of work (over two days) but I think it was worth it, since this is indeed a very important stag. Also, going over all the papers gives you some familiarity with each one that would be useful later on. I printed the first page of all submissions and tagged each one with its topic, already thinking who might be a good person to review it. (I also shared a pdf file containing only these first pages with my PC members, who found it useful to come up with their preferences.)

Technically, I did the assignment using an excel file with a row for every paper and a column for every PC member, containing the preference of the PC member. I initialized the file with the hotcrp-generated assignment but this didn’t matter much since I did go over each paper. It was more important to me that the right people review the papers than the load is exactly balanced. However, I did try to ensure that people that got a higher than average load did not get papers they marked as very low preferences. I handled a few papers that seemed out of scope (e.g., not theory, or P vs NP papers) separately, not assigning them to any PC member but making sure they each got a single review to doublecheck that they are indeed out of scope. One needs to invest some thought in assigning reviews, making sure that every paper has a sufficient number of appropriate reviewers, that papers on similar topics have overlap in the assigned PC members, and that the load is still approximately balanced. It took several passes over the list to get an assignment I was happy with, and even after finalizing it I needed to make (very few) changes due to PC members that forgot to put in conflicts.  I needed to write some simple scripts (which I’d be happy to share) to transfer the data between hotcrp and my excel file.

FOCS 2014 had a little over 270 submissions. Manually assigning papers for a conference with many more submissions would take of course proportionally more time.

At this point you’d also want to contact the chair of the previous conference to see which of the submitted papers are re-submissions of rejected papers. While the new submissions should still be independently reviewed, the work of the previous reviewers should not go to waste and needs to be an additional input in the discussions (I asked the paper chairs, see below, to look at those old reviews and see which parts are relevant, though there might be other mechanisms).

When going over the papers you might also notice some weird formatting errors, sometimes by people not too familiar with FOCS/STOC. I had one paper that was in a two column format and another where they didn’t compile it properly and all references and citations were question marks. In such cases I gave authors a short time (I think 24 hours) to replace the manuscript with a fixed version. (Of course I did not have to do this, but I don’t like the idea of rejecting papers based on technicalities; not because I owe the authors anything but only because my goal as a PC chair is to have the scientifically strongest conference possible.)

Step 6: The reviewing phase

The first couple of weeks of the reviewing phase were actually more relaxed for me as a chair. However, after two weeks I had a deadline for the PC members to post on the system a “shell review”. This is a review without scores but simply listing the names of their subreviewer(s) if any and their expected confidence for the paper. I think this was a very good idea. It allowed me to see early on what papers I might have trouble with (not enough high confidence reviews) and let me already start searching for additional external reviewers for these papers. It also forced the PC members to get an early start on going over the papers, finding subreviewers, and using the system. Partially because of this, many PC members started posting their full reviews early as well, rather than waiting for the last minute.

Essentially from the moment the review process begins, you need to monitor it daily, reading any new reviews that come in or comments the people make. Some of the things you need to do are:

• Solicit extra opinions: whenever I felt that a paper might not have enough high confidence reviews, or if I felt that a paper touches a particular area but doesn’t have some of the authoritative people in that area reviewing it, I solicited extra opinions myself. These might be full reviews or shorter opinions, but I wanted to make sure we have everything we need to make an informed decision. I sent about 120 such emails, and felt the inputs we received were extremely useful.
• Email the authors: Whenever a review mentioned a concrete issue with the paper that the authors missed: a bug in the proof, an uncited prior  work, etc.. I emailed the authors to get their response. Reviewers work under time should not be expected to be correct 100% of the time – sometimes the bug turns out not to be a bug (or to be easily fixable), the prior work turns out to be irrelevant. There’s nothing more annoying for an author than to have their paper rejected because the reviewer  made a factual error, but that’s not the reason I sent those emails. I just didn’t want to lose out on good papers, and wanted to make sure we are basing our decisions on the true facts. I feel that such emails work better than a formal “rebuttal” stage. First, I sent these emails throughout the process (sometimes multiple ones for the same paper) and so there was no need to wait for the rebuttal phase which can slow down the process. Second, I feel that it makes much more sense to send authors these concrete factual questions than to ask them to respond to the full review, the majority of which is the reviewer’s subjective opinions.  In many cases the reviewers were right, and indeed sometimes these emails caused the authors to withdraw their papers (saving us work). Sometimes the reviewers were wrong. Sometimes there was an issue with the paper, but the authors could fix it, in which case (after verifying the fix) we needed to decide whether the deviation from the original submission is too significant to be acceptable. But of course making judgment calls is our job. (Of course, aside from mailing the authors, if two reviews disagree on a factual issue you should point this out to the corresponding PC members and make sure they resolve it.)

As reviews came in, if one of the reviewers found a fatal issue in a paper (e.g., proof is completely wrong, result was known, or paper is a clear reject for some other reason) then, once I was 100% convinced  I shared this with the other reviewers so they don’t waste their time.

On the use of sub-reviewers. Omer Reingold posted a very thoughtful blog post on the advantages and perils of using sub-reviewers. On one hand, we absolutely need them: TCS is too broad for one committee to have the needed expertise in all areas. On the other hand, we don’t want to outsource the decision to sub-reviewers, since the whole point of a conference such as FOCS/STOC is to have results that have broad appeal across TCS and not just for the domain experts. Leonard Schulman  told me that he believes that a PC member should use sub-reviewers to teach her (or him) about the science behind the paper, but after doing that, make her own opinion. I agree with this sentiment, and this was one of the reasons I didn’t use hotcrp’s facility for external reviewers, as I wanted PC members to own their reviews, even if they consulted with others while making them. Indeed I initially considered not including a numeric grade at all in the default form PC members pass on to a sub-reviewer. I ended up including the grade, but still put that field last, trying to emphasize that this is in some sense the least important input we get from a sub-reviewer.

I didn’t attempt any “normalization” of reviewers’ scores (where one lowers scores of reviewers with high average grades or vice versa) since every grade has a semantic meaning which should be congruent with the text of the review. I did ask for clarifications if the grade did not seem justified by the text, and sometimes tried to get additional reviews in such cases.

If you have some free time at this stage you might want to get in touch with the publisher and make sure that the contract is signed and everything is all set for the camera-ready version. I know it seems far off, but you might have more bandwidth to deal with this at this point than later on.

Step 7: The electronic discussion phase

The electronic discussion phase is where you’ll make most of the decisions, and make sure you have gathered enough information for making the rest of the decisions in the physical meeting. The goals of this phase are to filter (i.e., reject) many of the papers so you can focus attention on those that might make it, and ensure you have sufficiently complete information on every one of the remaining papers.
The goals are that by the time of the physical meeting:

• A sufficiently small number of papers are in play so that you can have a meaningful discussion on each one of them.This was one of the most common pieces of advice I received from prior chairs. To get a sense of what “sufficiently small” means, if you use a schedule similar to mine, you will have about 14 hours of net discussion. Even if we make the optimistic assumption than on average a paper will be discussed for 6 minutes (including the overhead for transitioning between papers, which can also take a bit with people leaving and entering the room due to conflicts) that allows a maximum of 140 papers to be in play before the PC meeting. However, you don’t want to plan on that, since realistically some papers will be controversial and take a longer discussion, and often you will want to table a paper and raise it again at a later point. If you only have a 2-day meeting then to ensure things don’t become too rushed and chaotic, a rough target to shoot for would be to narrow the field to about 120 papers, out of which 20 or so are “conditionally accepted” and hence require very little discussion. (This is one point that I didn’t realize before chairing FOCS, and also places some barriers on significantly raising the number of accepted papers, unless one gives up on a physical meeting and the notion that PC members should have a global view of the program.)
• All the questions of science had been answered.

By the time of the meeting, we need to have all the facts straights about the papers in play. The questions discussed should not be whether the proof is correct, if X didn’t already prove it first, but rather we should only be discussing our subjective opinions on the importance of the result. If we start trying to figure out the proof of some lemma on the board in the meeting, or frantically email an expert, then we have failed in the electronic discussion. Despite my best efforts, we did end up frantically emailing an expert during the meeting, but to my recollection it only happened once or twice.

Reaching those goals is largely up to you as a program chair. You need to be constantly involved in the electronic discussion, making sure that no paper has outstanding questions that remain unanswered due to lack of attention. (In our conference we had an average of 11.3 comments per paper, but of course this varied widely between different papers.) A large gap in the scores between different reviewers is fine if it reflects a genuine disagreement on opinions, but you need to make sure that all reviewers agree on the facts. Often in this stage I emailed the authors and/or external experts as questions came up. For some papers that I felt could use an extra pair of eyes, I tried to see if there are PC members that have relevant expertise but were not assigned the paper and asked them to write an additional review (focusing on the issues raised by earlier reviews) or simply chime in the discussion. As mentioned, I also solicited extra opinions from experts. My goal was that for most of the papers that will end up being discussed in the PC meeting will have either a fourth review or at least an additional short opinion outside of the assigned reviewers. In fact, in many cases we had 2-3 such additional reviews or opinions.

Filtering papers. At this stage one needs to start removing papers from considerations so we can better focus our energies. While it is usually easy to identify a number of papers where there is a high confidence consensus on rejection and no unanswered questions in the review, very soon one gets to making harder decisions. PC members often feel uncomfortable with rejecting paper X when they see other papers still in play that they judge as weaker. However, one should remember that the decision to reject papers early is not about quality but rather about confidence. If we are very confident that paper X is a well-written, correct and novel result that does not rise to the threshold of interest for FOCS, then we should reject it early in the process, even if paper Y, that is we believe to be junk with 90% probability is kept alive, because of the 10% that it is a truly great result. What I did at this stage is look for papers where none of the reviewers were particularly excited, and then engage in discussion with the reviewers see if there is clear consensus that this doesn’t make the cut. If so, then the paper was tagged “reject candidate”. (Technically it was useful to sort papers according to their maximum score when looking for papers to suggest, but I always made the decision whether or not to propose a paper for “candidate reject” based on the actual text of reviews and discussion.) Then I emailed the entire PC that at date such and such all papers marked “reject candidate” would be turned into “reject” unless anyone object. We didn’t have votes since every PC member always had the right to keep any paper alive. I reminded PC members of some of the examples of great papers being rejected and tried to impress upon them the importance of going over the list of papers and making sure we’re not missing out on any great work. We had several such rounds.

Early acceptance. There may also be papers where there is a clear consensus among the reviewers that they are truly excellent and would be clearly way above the acceptance bar. In this case the papers can be marked as “candidate-accept”. We had about 20 such papers, which I think is a good number. Even so, these papers are not actually accepted before discussion in the PC meeting (though that discussion may be rather brief as they are typically uncontroversially excellent).

Problem cases. As the review stage proceeds you will encounter papers that for various reasons will take up much of your time and energy. Tolstoy said that happy families are all alike but every unhappy family is unhappy in its own way, and similarly every one of those problem papers is unique and your experiences would likely be different than mine. Nevertheless, there are some issues that seem to crop up again and again (though not all of those actually appeared in FOCS 2014)

• The super-complicated paper. A paper showing a great result but with a complicated and not terribly well-written proof that no reviewer can understand. This is a tough one. Many of the greatest papers in our field fell into this category, but also some junk papers that should not be published. First you’d want to make sure that the result is truly great, and that the paper does seem to contain genuinely new ideas. If this is the case, then just accept that this is going to be one of those few papers that will cost you and your reviewers a lot of work. One needs to make sure that you have several reviewers, both top experts in the field that can comment on the overall proof structure, and people that are willing to put in the effort of reading the paper carefully and making sure the proof makes sense. Back and forth interaction with the author can also save a lot of time here, and you should encourage the reviewers to not hesitate to bother the author with even the slightest question. (In one case I even suggested to the reviewers that they do a Skype call with the author, though they didn’t take me up on that. There were certainly several Skype calls between reviewers working together on some hard to verify papers.)Working hard is never fun, and you may be tempted to make a decision without this effort. Some people may say that such a paper should be rejected and the author should resubmit the paper with a better written proof. Other people may say that a conference is not supposed to verify the proofs anyway and if the paper claims a great result and clearly contains novel ideas then it should be accepted. I disagree with both views. I oppose rejecting such papers as indeed some of the greatest papers in our fields, ones that we are proud to have STOC/FOCS associated with and that have shaped entire fields of study, were of this form. Indeed, it’s often hard to understand if the proof is particularly badly written or is simply objectively hard, especially if the author is a relative newcomer to the field. Simply accepting such papers on faith is also not a good option. If a major result is claimed with an erroneous proof then that deters further people from working on this problem (and if someone later comes up with a correct proof we have messy issues of credit to consider).The goal is to reach the point where the experts either say that, while not checking line by line, they understand the overall structure of the proof and have confidence in its correctness, or they identify a major flaw. However, in the immortal words of another great author, “Sometimes you are winners. Sometimes you are loosers. We never can win against so many Poozers.”  Sometimes, even after dozens of back and forth emails with the author, and enlisting the best people to look at the paper, you simply cannot come to a conclusion if the proof is correct or not. In this case I think you have no choice but to reject the paper with a heavy heart. But at least I believe that this interaction would have a great positive impact on the readability of future versions of the paper.
• The causally related papers. These are two or more papers showing the same or related results. Not all of those are problem cases but they still deserve special attention. One common case is that Paper A proved result X and was posted online and then paper B proved the stronger result X’. While there are exceptions, typically in such a case you should not accept paper B without accepting paper A as well. After all, we want to incentivize people to post their results online, and also give credit to the paper that got the first result even if it had later been improved. Thus I would argue that even if X on its own would have been somewhat “below the bar” then accepting paper B should “pull up” paper A and cause it to be accepted as well. On the other hand, if the “delta” between X’ and X is not very significant, then accepting A and rejecting B is a perfectly fine decision.The reason I put this under “problem cases” is that things are not always so clear. Unfortunately, it’s not always the case that paper B properly cites paper A (or even cites it at all). In my advice for authors I told them that they should always err on the side of “overciting” a related work they know of, even if that related work is not yet formally published, but it still happens that authors neglect to do so.  Whenever you have two related submissions, you should make sure you are very clear on the timeline of who came first, checking the arxiv/eprint/eccc/etc.. trail as well as contacting the authors to get clarifications. Some authors use the phrase “independent work” in a rather confusing manner. If the paper B was done after paper A was posted online (or otherwise came to the author of B’s attention) then it is not independent, even if it uses completely different techniques. There are even more complicated issues. What if paper A wasn’t even submitted to your conference? What if paper A wasn’t posted online but was still somehow available to the authors of B? What if it’s not clear who came first? There are not always easy answers to these questions, but you’ll be in much better shape if you get your facts straight on who did what first before the PC meeting.
• The reviewer with an agenda. It’s perfectly fine for some reviewers to have strong opinions about a paper, but what you have to watch out for is when the reviewer’s opinion is not really about the particular paper but about a more general theme (whether they strongly like or dislike area X, or papers of the form Y, or feel that TCS should move in the direction Z). Again, it’s also OK to have such opinions and factor them into your review, but you should be clear about that, and make sure that you are not completely ignoring the actual paper to make the general point. In many cases such reviewers do not really cause a problem – if a review is a clear outlier and doesn’t make a convincing case then you can simply ignore it. The problematic case can be when the paper is in a topic where the PC has little expertise and the reviewer is a very well respected figure in that area. One needs to watch out for such cases and try to enlist additional reviewers if needed.
• Controversial papers. Some papers will be controversial because judging them requires balancing competing considerations. For example, what do you do with a paper that proves a strong math result but one with tenuous CS applications? Or a paper that has significant practical impact but is not so interesting from a theory perspective? Or a truly badly written paper that proves a strong result? Or a paper that is a pleasure to read and you feel you learned from, but the actual novel results in it are quite modest? Or a paper that is a collection of nice results on a related topic but no single one of them is very strong? Or a paper that introduces a potentially interesting new model but is technically trivial? Or a paper that solves a very hard problem but one that only a handful of people would be able to follow? Or a paper that ties TCS to some other field X in a way that seems natural to us but is not interesting to the people actually doing X?In some sense evaluating every paper requires answering some of these questions, but the true mark of a controversial paper is when the answer seems obvious to most people looking at it (as I’m sure is the case with many of the questions I listed above) but different people think that different answers are obvious. One key for dealing with a controversial paper is to try to avoid broader philosophical discussions that are not directly relevant to the paper at hand. In particular if a paper is going to be rejected for reason X, you shouldn’t discuss whether or not it should also have been rejected for reason Y. Sometimes these broader discussions are unavoidable, and in this case you should try to do them by a PC-wide email thread before the PC meeting, as they can easily suck a lot of time and energy during the physical discussion.

While others might disagree, my personal philosophy for FOCS/STOC papers is that you should look for reasons to accept papers as opposed to reasons to reject them. This might sound like a meaningless “feel good” statement, but it actually does have an operational meaning. It means that when evaluating reviews my focus is on what is good about a paper rather than its flaws. It’s often easy for reviewers to hide behind objective flaws (e.g., the paper is badly written, they didn’t cite work X, there is a fixable bug in Lemma Y, went over the page limit if you have one, etc..) and use phrases such as “this paper is unacceptable in its current form”. I often pushed back on such reviews, asking people to focus on what is their subjective opinion on the paper’s merits (though of course we do want to know about the flaws as well, and make sure they are fixed).

I should say that many people disagree with me and argue that writing quality should be a significant factor in the decision, since the reader can get more from a paper (or a talk) if it’s well presented, and accepting poorly-written papers may lead to low standards of presentation in the community. I am not unsympathetic to this argument, and I think that regardless of whether we factor this explicitly, better-written papers will (and should) always have an advantage. However, I am willing to cut a lot of slack to the paper if the result is truly great, and I want reviewers to make the effort to determine whether or not that’s the case. We also need to remember that there is no single FOCS/STOC community. Different authors come from different communities with different cultures and styles. What we perceive as a “badly written” paper could be the result of a newcomer or outsider to the field not sharing the same context as we do. Nonetheless I would agree to set a higher standard for authors that have had several FOCS/STOC papers and should know how to write their submission properly. For others, “shepherding” (where we accept the paper on certain conditions of improving presentation) might be in order so the paper can realize its full potential for impact.

Another corollary of this philosophy is that there should actually be a reason to accept the paper. It’s not enough that there’s nothing wrong with it – a paper in FOCS should have something about it that is exciting and worth announcing to the broad TCS community. I believe that FOCS doesn’t need any “filler papers” and so I also often pushed back when PC suggested the paper should be “accepted if there’s room” even though they don’t personally consider it as an extremely exciting result, sometimes using the argument that “some papers must be in the bottom half of accepted papers”. While that is indeed the case for every particular ordering, I think that FOCS is a diverse and broad enough conference that every paper should be in the top half in the eyes of its champion. For papers in areas under-represented in the PC I would be fine if the true champions are only the external reviewers, but I believe that if a paper in a core area such as algorithms or complexity is strong enough for FOCS, then it should find at least one true champion in a 22-strong PC. Madhu Sudan commented that another justification for this philosophy is that a PC never rejects a paper – it either accepts it or tables it for discussion in a future conference. Indeed “accept/reject” is probably not the right terminology; perhaps one can phrase the decision as to either “broadcast” the paper to the community or not.

On “fairness”. Fairness is an issue that often arises in discussions of competitive conferences. This is not surprising, as the PC is charged with curating an exciting program based on their subjective tastes, but their decisions can have an impact on authors’ careers. Different people mean different things by “fairness” but I think we can all agree that PC’s should not discriminate authors on the basis of gender, race, university affiliation, or other such attributes. I understand the intuitive appeal of trying to achieve “fairness through blindness” with practices such as anonymous submissions, exclusion-based conflicts etc., but I think the approach of “fairness through awareness”, with disclosure-based conflicts and trying to be aware of biases might work better. In particular, one thing I did not do but could perhaps make sense is to consider assigning an extra review for papers of authors from underrepresented groups or first-time authors as a way of countering the possibility that the work would be dismissed too easily due to some hidden biases.

Step 8: Preparation for the meeting

The PC meeting is where all final decisions are made. It is also one more chance for you to screw up, which is not that unlikely since the schedule will likely be packed without much room for errors.

As one way to prepare toward the meeting, several weeks ahead of it I appointed for every paper that was not already rejected a paper chair. This person, who was usually (though not always) one of the reviewers needed to guide discussion between reviewers towards clarifying which points are in consensus and when there is disagreement. I also asked paper chairs to see whether they feel that the paper could use a look from any particular people from within or outside the PC. He/she also needed to tag the paper’s topic (we used a system of “main topic.sub-topic” such as “alg.online” or “complexity.derand”, which makes it easier to bunch together related papers when sorting). If paper X is particularly related to paper Y, then the chairs of these papers needed to make sure there is some discussion comparing them (typically I tried to ensure such papers would have the same chair). While paper chairs were generally great, you also have to go over each and every paper and make sure these things are actually done. Finally, the paper chair needed to write a summary comment in the system, which contained a 2-3 sentence description of the paper’s “claim to fame”, its mains pros and cons, short quotes or summaries of the viewpoints raised, and the ID’s of papers that it is particularly related to. I wrote a script that automatically converted these summary comments (as well as other data such as scores and confidence) into beamer LaTeX slides that we used at the PC meeting.

Toward the PC meeting you need to come up with the list of papers that will be discussed and the order that they will do so. Following Omer Reingold’s lead, before the meeting I asked each PC member to send me four* lists of papers:

• “Must Accept” papers – these are papers that were not already tagged as “candidate accepts” but for which the PC member would go home feeling truly disappointed if they were not accepted. While I could not of course guarantee that they will be accepted, I did guarantee that all these papers would receive ample attention and discussion. (There were relatively few such papers, since most of the ones in these category were already marked as “candidate accept”.)
• “Advocate accepting” papers- these are papers where the PC member would champion acceptance even if they don’t feel as strongly about it as the “must accept” papers.
• “Special” papers- these are papers that the PC member feels are “special” in some way and wants to make sure they don’t “fall through the cracks”.  Papers can be special for many reasons. Perhaps they are on a topic that is not well typically well represented at FOCS/STOC (or in the PC). Perhaps they have a mostly conceptual contribution that is harder to evaluate. Perhaps they can contribute something unique to the program in another way. I wanted to make sure these papers are discussed and encouraged PC members to note such papers (as well as the reason for this tag) even if they don’t personally support accepting them.
• “Controversial” papers – papers that are controversial for any reason, and hence would need extended discussion. (Assuming we didn’t manage to finalize the decision on them in the electronic discussion phase.)

* I actually only asked for three lists at that point in time, since we’ve already tagged controversial papers earlier in the process.

One more piece of preparation is to order food for breakfast, lunch and coffee breaks, as well as make dinner reservations. Don’t skimp on the food. These people are volunteering dozens if not hundreds of hours of their time, and making the effort to come, so the least you can do is make sure they are well fed. Of course good food doesn’t have to be expensive (I used Clover food labs for lunch and I believe most people were very happy with it). You want everything to be delivered to the place where you are deliberating to ensure that you don’t waste time going out and people can keep discussing papers during the lunch and coffee breaks. You should take them out to dinner in the night between the meetings (which may actually be a better atmosphere to discuss some controversial or “philosophical” issues), and I also took those who didn’t need to catch a plane to beer after the meeting ended. You’d want to check with the local chair the procedures for PC reimbursements. You can ask your PC members if they might be able to cover their travel and hotel expenses from their own sources. Usually several people have no problem doing so, and if even one of them does, it will may well be more than the total cost of food.

Step 9: The physical meeting

Running the physical meeting requires a different skill set than the rest of the process. I personally felt that I didn’t do a super great job at it, primarily because I seem incapable of stopping what I feel is an important discussion because we should keep to the clock. But I was saved by good preparation and so we did manage to complete all we wanted to do, even if it was a bit rushed at the end and we didn’t have the buffers I was hoping for.

Some practical tips. You should assign a person to mark the decision for each paper (accepted, rejected, tabled, etc..) as well as add notes if any are needed. You should also assign a different person to keep track of the time. You’d want to use some app displaying a timer on the screen. I allocated a total amount of time for discussing each paper (if I recall it was between 4 to 6 minutes depending on the amount of expected controversy) but in retrospect I would have separated it to 2 minutes for someone to present the paper and another X minutes to discuss it. What often happened is that the person initially presenting the paper used up almost of all of the allotted total time for it, and then we had to add more time for the discussion. Another tip I got from Omer Reingold: sit in the middle of the table, so you can better control the discussion.

Order of discussion. Given the tight schedule, the order you discuss papers can make a big difference in the final result. Perhaps the simplest approach is to simply sort papers in descending order according to the average score. However, there are two issues with this approach. First, with such an approach small differences in the scores (which are completely meaningless) can make a huge impact on when the paper is discussed. Second, when you follow this approach you will naturally accept many of the initial papers, but at some point the PC may feel anxious about accepting too many papers and this may cause a paper to be rejected even if it didn’t need to be. Some people suggest to counter this by alternating the order from time to time, and discussing papers from the bottom of the list. But this seems to me artificial and also potentially wasteful. If you run out of time, then those bottom papers might be precisely what you want to skip.

Following Omer, I used the lists above to set the agenda for the first day. We discussed the “candidate accept” papers first, then those on the “must accept” lists, then the “special” papers, the “advocate acceptance” papers, and the “controversial” papers. As a rule, we did not reject any of the “must accept” or “advocate acceptance” in the first day – we either accepted them outright or tabled them for discussion in the next day. Within each list I first sorted the papers according to how many people advocated for them, and as a secondary sorting I did use some function of the scores. (If I recall it was actually a sum of squares of the scores..) If two papers were listed as related to one another and in the same list, then I made sure they are were discussed together regardless of the score. (One nice feature of hotcrp is that you can input to it an arbitrary order and then sort the papers according to it; the slides were also sorted in discussion order.)

On the night of the first day I gave the PC members homework to see which paper they advocate for raising for discussion in the next day. I stressed to the PC that no paper (including those tabled in the day before) will be discussed if a PC member did not explicitly advocate for it to be discussed.  (After all, if no one is going to support a paper to be accepted then what is the point in talking about it.) The next morning I generated the order based on that. (This required a bit of scripting under time pressure and I did have a bug that delayed us by about 20 minutes.) At the lunch time in the second day there was a last chance for people to include more papers in the discussion list. I also had a list of papers that I would have liked to discuss if there was time even though no one advocated for, but we didn’t end up having time for them.

As mentioned, I had a script to automatically generate a summary slide for each paper that we used in the discussion which we projected on a screen during the discussion.  I thought these slides were very useful way for PC members who were not assigned the paper to get a quick summary of its status. Ahead of the meeting I generated for each PC member a pdf file with the slides (in discussion order) of all the paper for which they didn’t have a strong conflict. (This was the script I was most nervous about and checked the most times, since I was bypassing the system and hence could have potentially introduced a bug leaking the summary of all information about a paper to a person having a strong conflict with it. In the main pdf file we used for discussion, the script generated for papers with a strong conflict first a slide that only contained the paper name and names of people with conflicts so that when it is displayed they’ll know to leave the room and we don’t accidentally project it with all information while they’re still in the room.)  I think this pdf was useful for PC members to both go over the papers before the meeting and of course during it to keep track on their machines on the paper under discussion.

Voting. There are several schools of thought regarding voting. One is that you should discuss until you reach consensus, but that can of course take forever. Another approach can be that you should only accept a paper if you get a strong majority, since the paper’s advocates should be able to make the case for the PC. On the other hand, you can also say that you should only reject a paper if there is a strong majority, since it’s enough that some fraction of the community is excited about the paper. I ended up going with a simple majority, but encouraged everyone in the PC to vote, and feel free to take into account in their votes not just their own opinions on the paper, but also their judgment on the sense of others’ excitement. I didn’t mind if a decision was made on the basis of a 12 vs 11 vote. What I did mind is if the vote was, say, 4 vs 1, and in this case I insisted on more discussion and voting and there was significant participation from the PC.  As chair, I never voted.

Number of papers. Following the advice of some previous chairs, I didn’t set any fixed number of papers to accept, and in fact didn’t even state a range of numbers.  The rationale is that once there is a fixed budget it becomes a “zero sum game” and PC members might feel they need to argue against paper X to make sure there’s room for the favorite paper Y. I just asked the committee to have high standards. There’s considerable logistical flexibility and we would have managed even if we had accepted more papers than usual though we ended up accepting a little less than the average.

Some issues to watch for.  There are some issues one needs to watch in the PC discussion (apart from running out of time). One scenario that I was warned against but fortunately didn’t experience is that the discussions become “partisan” with each PC member only fighting for papers in his/her own field, instead of a collaborative process where members see their role as educating the rest of the committee on the positive and negative issues in the papers. I think having no fixed budget for papers helped in avoiding this, as well as the large control PC members had on the order and content of the discussion via the “must accept” and “advocate acceptance” lists. Also, my PC members were just awesome people. Another issue one needs to constantly watch (also in yourself) is the temptation to try to be consistent. If the PC rejected paper X, you can’t use that as an argument to reject Y even if you think Y is weaker than X. Otherwise, if you lost a single argument and one of your favorite papers was rejected you would then be arguing to reject all remaining papers. Similarly, if a PC member feels the PC accepted a weak paper, it doesn’t mean that from this moment onwards it should accept all papers. You and your PC need to accept the fact that the PC’s decision cannot be consistent with any particular person’s opinions, including your own. This is sometimes hard. When phrasing the rejection letter, I was thinking of the authors of some of those papers which, if I was a dictator, would have been accepted. The only time you need to make consistent decisions is when two papers are casually or temporally related (as mentioned above).

Your goal is to make all accept/reject decisions in the PC meeting. I feel it’s good to send the reviews together with the decisions, and so I told my PC in advance that they would have only 48 hours after the meeting to make any edits before I will send both decisions and reviews to the authors. Sometimes people assign a PC member to add a “summary review” for the author explaining a summary of the PC discussion. However, I skipped this for several reasons. First, not all papers are discussed in the meeting. Second, in many cases the summary would repeat things that are already in the reviews. Third, I overworked my committee members enough during the process and didn’t want to assign them any additional task if I absolutely didn’t have to. In very few cases (for example, if it’s a paper we really wanted to accept but just couldn’t because despite heroic efforts no one could get enough confidence in the proof) we tried to make sure the rationale for decision is absolutely clear from the reviews; in some cases I might have written to the authors separately.

In the email to authors of accepted papers, you usually ask to confirm that the title and authors names and affiliations are correct. I also asked them for the URL to a freely available online full version of their paper which I put on the conference website. In addition I asked them to send to me a response to the reviewers comments, detailing which ones they implemented and which ones they didn’t and why. I think that one (perhaps even the main) disadvantage of conference as opposed to journals is the lack of a process of revisions between author and reviewer which often greatly improves the presentation. While we don’t have time for such a full process, I felt that even adding this “half iteration” can benefit. Also, if authors decide not to implement a comment, I think it’s good that the reviewers, who have spent significant time and effort, see that it wasn’t just neglect but rather a thoughtful conscious decision.

You will get some emails complaining on the decisions. As hard as you try, sometimes the authors will point out to genuine mistakes on the part of reviewers (though emailing authors during the process greatly reduces this). My view is that, like in soccer matches, the decisions are final and are not changed even if you find new information. For example, after notification I found out that one of the reviews for one of the rejected papers falsely claimed that some of the results were known before. While this wasn’t the reason for the decision (which I didn’t revisit), I felt (and still feel) bad about this. (When I looked back at the discussion for that paper, I saw that I made a comment asking the reviewers whether we should check that claim with the authors, but forgot to follow up on this; this was one case I regretted that I wasn’t even more “trigger happy” with emailing the authors.) One exception to that is if you discover that authors behaved unethically in some manner (e.g., neglecting to cite prior work that achieves the same or close results) in which case I think it’s sometimes justified to rescind the acceptance decision.

 

Step 10: Producing the proceedings

Once this phase is done, you would start being in more frequent touch with the contact person for the proceedings’ publisher and make sure the author kit is sent out. (The contract should have been finalized several months before then but some details such as the number of papers accepted are of course only known after decisions are made.) Amazingly, even in the age of electronic proceedings, it turned out that we still pay IEEE per page. For this reason, I went with the traditional (and in my eyes incredibly ugly) two-column template. In any case, I think these days the arXiv version is much more important than the proceeding version. We saved some more money by skipping the USB sticks. One more tip is that the deadline to the camera ready version should also include a time and not just a date. Don’t assume that things came in, and keep in constant touch with the contact person to make sure that everything is in order. In particular, make sure to mail the authors of accepted papers soon after notification their “author kit” and instructions on how to prepare the camera-ready version.

As information comes in (updates to authors’ names, affiliations, titles, URL, or as you make and change scheduling decisions) you’d want to update your website. I had scripts to use information from hotcrp and tables I maintained to generate the list of accepted papers, and later also the web program, latex for the printed program, and data for the mobile app.

Usually, you also need to set the schedule by the time the proceedings are finalized since it is often printed in presentation order. If you want to buy yourself more time  to set the schedule, you can do as I did and order the papers alphabetically instead of by sessions. You’ll also need to write the foreword with acknowledgements, awards, etc..  It’s best to get a copy of that from the previous chair and modify it. (Once again a script is useful to extract the list of external reviewers from the system’s data though you’ll of course need to go over it to fix omissions, repetitions, and misspelling.)

Step 11: Best paper and special issue decisions

These decisions don’t need to be made at the PC meeting. For the best paper and best student paper, we quickly came up with a short list of candidates. These papers need to be vetted further, both for correctness and for importance, so for every candidate I appointed a “handler” whose goal was to collect even more opinions (both on significance and correctness) on the paper from luminaries in the field. I think this is particularly important because those papers might have not had significant discussion in the review phase, being “obvious accepts”. It’s also fine to decide not to award the best paper or best student paper awards if no candidates particularly stands out.

I appointed two great PC members as editors of the special issue and put them in touch with SICOMP. I left them the task of selecting the papers (with input from the PC).

Step 12: Setting the schedule, mobile app

Before making the schedule you need to talk to the local chair and the SIGACT/TCMF chairs to make sure you are aware of all physical constraints (number of rooms, capacity, schedule) as well as possible other events (e.g., invited talks such as Knuth prize etc..) that you need to account for. You might want to also be in touch with the workshop/tutorial day chairs and make sure the conference website links to their information. I also doublechecked with the local chairs that there will be breakfast as well as cookies etc.. in both morning and afternoon coffee breaks, and of course beer in the business meeting. (I get awfully disappointed if there’s not enough free food in a conference J.) Depending on the location, you might want to end earlier on the last day if most people will take an afternoon/evening flight home.

Since we accepted somewhat fewer papers than usual I was tempted for a while to try to make the conference single session, but it would have resulted in a crazily long and dense schedule. Instead I used this flexibility to have reasonable breaks (I am a big believer that a lunch break should be at least two hours) and to have all the special issue papers in a single (non parallel) sessions in the beginning and end of each day. I personally greatly enjoyed those sessions, since each paper was on a completely different topic but you could still see some connections, and it felt to me like they truly embodied the spirit of FOCS/STOC.

I don’t know of a particularly clever way to make the schedule. I started by tagging the papers according to topics that will become thematic sessions, and then played around copying and pasting an excel file until I got a result I was happy with, that didn’t have too many conflicts and seemed to have a good balance for each day. Nonetheless, at the actual conference there was more than one time that I discovered that I scheduled in parallel two talks that I wanted to go to.

You also need to find out who in your PC is going to the conference and set session chairs.

One idea I had but ended up not pursuing because of lack of energy is to ask the PC to prepare for each paper in the program a “blurb” – a 2-3 sentence description of what the paper is about and why it’s exciting (like the blurbs in back of books). I thought this kind of document might be useful for people to get a sense of what talks they’ll want to go to.

Setting up a mobile app. I had a mobile app for the conference which I think was very successful. It allows participants to see the agenda, including what’s right now, as well as mark in advance the talks that they want to go to and get reminders X minutes before the talk. This is particularly useful in conferences with parallel sessions. The app also allowed one to see the actual papers (which is particularly useful for tablets; there was a web version as well). We used Guidebook.com, which I’ve seen used successfully in the ICM 2014 conference. It wasn’t super cheap but I also didn’t think it was too costly given the utility, and I found their customer service to be extremely helpful. If you are a chair and want to use them for a conference, I’ll be happy to talk more about it, as well as share code and data on my project.

Step 13: The conference itself

At the conference itself you can mostly sit back and enjoy the talks. It’s the local chairs that do all the work now. Perhaps you can (like I did) send a reminder to the session chairs of each day. For the business meeting you need to prepare a short presentation with some funny statistics.

Some conclusions and suggestions for the future

On the whole, there are a lot of things I liked about chairing FOCS. The time-limited aspect of it brings out a lot of energy and everyone involved, including authors, reviewers, and PC members, generally rises up to the challenge. This makes things much more efficient. I think that even if I stay on as a JACM editor until I’m 100 years old (a frightening thought..), I would not handle as many papers as I did for FOCS 2014. Sure, not all conference reviews are high quality, but then again the same holds for journal reviews. But perhaps the thing I like the most about a conference as opposed to journals is the ability to make the decisions jointly in a meeting with 22 fantastic people. I am much more comfortable with that than with having to make the decision essentially on my own as a JACM editor. As I mentioned above, I do think journals have their own advantages as well. In fact, I think one good idea would be for some journal to automatically accept any STOC/FOCS paper and have the paper jump straight into the revision stage. (This does not happen today even for best papers or special issue papers.) Hopefully this would result in timelier journal versions.

Overall, I think that when executed properly, the selection process of FOCS/STOC works pretty well (despite the inherent subjectivity and randomness in this process). That said, there can always be improvements, and I’m looking forward to any innovations future chairs come up with. My biggest issues are with the event itself, as I detailed in this blog post with Omer Reingold.

[Minor updates on 3/11/15]