[Yet another personal post in our series]
STOC Festival Design: Improving interaction and fun factor; reducing information overload.
Sanjeev Arora
How can we increase the value added by a conference in today’s information-rich world, when papers have been available on arxiv for months to the experts in that area?
These are some personal thoughts (ie I am not representing the committee or SIGACT).
First, I wish to make a plug for poster sessions at STOC: all papers should also be presented at an evening poster session. If you missed a talk, you can get the 2-5 min (or longer!) version at the poster session, tailored to your level of prior knowledge and speed of comprehension. (Remember, theory says that 2-way communication is exponentially more efficient than one-way!) Poster presenters —often students and junior researchers—will get a chance to meet others, especially senior researchers. Ideas and email addresses will get exchanged. (Currently I talk to approximately zero students at the conference— certainly, nothing facilitates it.) Also, different coauthors could present the talk and the poster, which doubles the number of people presenting at the conference.
Second, conferences should do a better job to help us navigate today’s sea of information. (As Omer notes in his post, we can decouple the “journal of record” role of STOC with the actual program of talks.) The current format of 95+ talks of 20 min is very fatiguing, and it is hard to figure out “What to do if my attention span only allows N talks?” Arguably, this question can be answered by the PC, but that signal is deliberately discarded and hidden from the attendees. One way to reveal this signal would be to schedule talks of different lengths. For example, with 130 accepts one could have 8 talks of 20 minutes in plenary sessions, 48 talks of 20 minutes in two parallel sessions, and 74 talks of 5-minutes each in two parallel sessions. (And all papers would also be presented in poster sessions.)
Benefits: (a) Allows substantial increase in number of accepts to 130 while staying with two parallel sessions. (b) May lead to a less risk-averse PC (ie more diverse conference) while maintaining a very high-quality core. (c) Attendees get to tailor their consumption of content. (d) A 5-minute talk is still enough for the presenter to give a sense of the work and publicize it. Each attendee gets exposed to ½ of the overall program instead of a 1/3rd; this is efficient use of their attention span.
Possible objections: (a) Effect on tenure/promotion. (b) Noisiness of the signal (c) Authors are worse off.
I think (a) will become a non-issue. If today’s tenure case has X STOC papers, tomorrow’s might have X/2 papers with 20-min talks and X with 5-min talks (b) Yes, PCs are 100% fallible, but weigh that against all the benefits above. If we don’t believe in PC judgement we might as well disband STOC.
For (c), let’s do quick a Pareto analysis. The comparison plan on the table is 95 accepts: 8 plenary talks + 87 talks of 20 min in three parallel sessions. (We need three sessions because of the substantial plenary component being added.)
With 130 accepts the turnout will be higher; perhaps 25% higher. Authors are trying to maximize the number of people exposed to their paper. The basic math is that ½ of 125% is roughly twice of 1/3rd of 100%. We’ll see that all authors are much better off in this proposal, except those whose paper had a nominal “rank” of 57-95 in the PC process, who both gain and lose.
Rank 1-8: Somewhat better off (125% vs 100%)
Rank 9-56: Significantly better off (62% vs 33%).
Rank 57-95: Gain and lose. (An audience of 62% instead of 33%, but 5 min talk instead of 20 min.).
Rank 96-130: Significantly better off. Their paper gets into proceedings, and they get 5 min to pique the interest of 62% of the audience (without waiting half a year to resubmit).
Every ex-PC member will tell you that papers that end up in the 3rd category were equally likely to be in the 4th, and vice versa. Knowing this, rational authors should prefer this new plan. It makes smarter use of a scarce resource: attendees’ attention span.
Windows On Theory 
Commenters, please do not overlook an important line:
The comparison plan on the table is 95 accepts: 8 plenary talks + 87 talks of 20 min in three parallel sessions.
Do not compare to the current format of 90-ish accepts in 2 parallel sessions, which doesn’t fit in a theory festival.
I generally like the direction of the proposal quite a bit. Let me stress 3 points.
(1) The current format discourages us from talking to each other as we have to skip talks to do it. It’s unfortunate: You need to sneak out on a someone’s talk feeling all guilty just so that you can talk to somebody else. Lunch breaks are short and hectic.
(2) Five minute talks are actually pretty good! Speakers are way more conscious of the time constraint and put serious effort into their slides. If I could choose between hearing a 5 minute talk and a 20 minute talk on the same topic outside my area, I’d generally go see the 5 minute talk. Twenty minutes is actually a very tricky length. It’s more than enough time to lose 95% of the audience—yet often not quite enough to please the remaining 5%. From the perspectice of a speaker, having 5 minutes in front of a big engaged audience is way better than 20 minutes in front of a small tired audience suffering in silence.
(3) Well attended poster sessions are great. It’s where you go to talk to each other. See (2). Following up on those papers whose 5 minute presentations you liked most is also fun. Past STOC poster sessions gave a bad impression because they were poorly attended. To make them well attended you need to include lots of posters corresponding to high quality submissions from the main submission track of the conference (as the idea of the new proposal).
For these reasons, while initially my opinion was the opposite, I’ve come to like the NIPS format a bit better than the traditional STOC/FOCS format. At any rate, I think that many people greatly overestimate the potential negative impact of trying out the proposed changes.
I like the proposed changes and would love to see them implemented. Especially the poster session idea. I think it is particularly important for graduate students — when I was in grad school I had posters (sometimes even more than one) at every STOC poster session and enjoyed this a lot (especially when STOC was a part of FCRC when the attendance was higher). As you say, with interaction presenting your work is much more efficient. It was sad to see the poster session idea abandoned recently.
The 5-minute talks are a great idea too, but might be a little more controversial. So trying it out would be great, but whether this will stick is not so clear. Technical details are already hard to fit into a 20-minute talk so with a 5-minute talk it seems almost impossible to go really technical. This might be what makes a difference between theory and more applied conferences like NIPS which use similar format. I think some of the commenters on Boaz’s post (Raghu, Konstantin) already mentioned this. A more subtle issue is that reduced amount of exposure at STOC/FOCS might drive some subfields away to their more specialized conferences — instead of getting a 5-minute talk + poster someone might prefer a regular talk at a more specialized venue. Finally, it seems inevitable (even if discouraged as I think was proposed in one of the original posts) that decisions made by the PC regarding the allocation of slots will eventually make it into job applicants’ CVs (as they do for NIPS). Whether this is good or bad is hard to tell.
My personal take is that in addition to whatever the format of the new joint STOC+FOCS conference is colocating it with other conferences seems to be a relatively easy and doesn’t require commitment to any serious changes: http://grigory.us/blog/stoc-focs-proposal-colocate/
Grigory:
Thanks for the feedback. For STOC vs a specialized conference, authors have presumably already done that calculation, and concluded that they have a result that they want to publicize to a broader theory audience. 5 min to an engaged audience of 200 + poster may well be an acceptable tradeoff. Or maybe the 5 min talks could have the option to opt out of the proceedings, and thus not lose copyright. Many tweaks are possible.
This proposal is just an interesting design alternative that the community should look at.
More generally, the community should discuss how to enable means of keeping us afloat in the sea of information. 130 talks with no differentiation is like a live version of arxiv (minus the browsing option).
A possible middle ground might be to have 20-minute talks + posters for each talk. This requires only slightly more time (extra 2-3 hours) for the poster session. I’ve been to ICDE which uses this format. An upside is that it requires only a very minor overhead from the participants. Since they already have their 20-minute talks prepared if they don’t want to prepare the poster separately they can always just print out their slides onsite. Of course, some people will put effort into making great posters, but having a backup option with no overhead should probably encourage people to participate.
Grigory I didnt understand your 2nd comment. There is a huge difference between a schedule with 20 min talks and one with 5 min talks. The former must necessarily take 4 times as long, or be in many parallel sessions.
I am indeed proposing that *all* STOC papers should also be presented at poster sessions, regardless of how long a talk they are allocated.
Yes, sorry I didn’t explain that having more parallel sessions is indeed one possible way to make it all fit. Parallel sessions seem more acceptable given an extra option for a poster presentation. When it comes to attendance, I think that the old STOC poster session actually helped increase it. Students who didn’t have STOC papers but just presented their recent work at the poster session had a stronger incentive to come than they would otherwise. So I think that allowing contributed non-refereed posters from the participants might be a good idea too.
The reason why I didn’t suggest the exact format is because it is not really clear to me how the scheduling is going to work at the Theory Festival so I was thinking that this is still being debated. If I remember correctly then I believe the original proposal was to have a single Theory Festival that combines papers from STOC and FOCS and lasts for 4-5 days and your suggestion seems to apply to STOC only. Both sound better than what we have now so trying either one would be great. When it comes to 5-minute talks then I am personally on the fence, but trying it once definitely can’t hurt and will give us a better idea of whether it works for the theory community or not.
I do not see how one would use 5 minutes, given that most talks are highly technical. In the 20 minute format, typically the first 5 minutes are spent introducing enough of the background so outsiders have an idea of the state of the field before the result. This often takes as much as 10 minutes. I think this is an excellent use of time, especially for grad students, who get exposed to polished introductions to subareas.
We might as well make good use of technology, and have polished 20 minute presentations available online–perhaps before the conference.
Another alternative would be an expansion of the tutorial idea: if there is a group of accepted papers that address the same subtopic, the 5 minute talk could work with a 15 minute overview introducing the papers. The program committee could ask a subset of the authors/reviewers/PC to do it: it could become a prestigious thing to have done, and would benefit the community.
I’m always surprised to see so many people have preconceived version of “how a 20 minutes talk should look like”. The usual version they keep repeating is that “the first 5 to 10 minutes is devoted to an introduction for the general audience and the rest is for the technical part”. I wonder whether this view is the reason that so many talks are, well… not very good.
The truth is that giving a good talk is an “art”, there are no rules, and no single scheme is suitable for two different papers. There are good papers with no interesting technical parts, and there are papers that are so technical that it’s useless to even get a hint of the details. There are endless ways to structure a good talk. And a good speaker may have a brilliant five minutes talk, while for a bad one 20 minutes will just be a complete waste of time.
Ben, you have it exactly right.
[This comment is a general one about the STOC 2017 Theory Festival ideas, not specific to Sanjeev’s proposal.]
I have finally had a chance to read through all the blog posts and comments about a STOC 2017 “Theory Festival”. I am thrilled to see our community addressing these important issues in such a thoughtful and collective manner, with a serious plan for implementation — many thanks to Boaz, Omer, Piotr, Sanjeev, Tim, and the SIGACT/TCMF leadership for taking this on!
I agree wholeheartedly with essentially all of the goals of the working group and many of the sentiments that have been expressed:
– creating a “must-attend” event that will bring the entire TCS community together
– a focus on the audience/attendees in structuring the event
– the importance of conferences and the design of their programs in helping reduce “information overload”
– an expanded plenary program, which ensures that we all are hearing about what is happening in other areas of TCS and related fields, accompanied by more
specialized parallel sessions for interaction within subcommunities
– more time for informal interaction, including a poster session
– not dramatically changing the current STOC paper acceptance+publication process, so as not to break something that is working reasonably well (albeit not perfectly)
It is natural and healthy that we have many different ideas on the details of how to implement a vision like this, and I think that many of the proposals that have been discussed will be a big step forward. Still, let me share some thoughts on the specifics:
– Colocation. Several times in the blog comments, colocation of a number of theory conferences has been suggested as a way to help create the theory festival, and then dismissed as being infeasible. Why is it infeasible? CCC and EC colocated with STOC in 2010, and we’d only need a couple more (could be SoCG, LICS, RANDOM-APPROX, TCC, PODC, …) to have a theory festival. I would imagine STOC being in the middle of such an event, ideally with all talks being plenary (even if this means a reduced number of acceptances), and the others being run in parallel before and after STOC. Ideally, the notification deadline of STOC would be before the submission deadlines of the other conferences so that authors do not need to choose whether to try STOC or submit to a specialized conference. The advantage of this approach is that (a) we reduce rather than increase the travel burden on the community (if we really are going to draw people who currently don’t attend STOC regularly, then the STOC theory festival will be an extra week of travel for them – unless they can eliminate something else from their calendar), and (b) many more people will have a paper accepted to justify the travel. More discussion of this idea is in my post during the 2010 “Future of STOC” discussion: http://futureofstoc.blogspot.com/2010/06/salil-vadhan.html
[As some have noted, colocation is something that can be done/added in the future, if 2017 is too soon to arrange it and/or we need to first prove the success of the event to attract others.]
– The plenary program. As mentioned above, I very much support having a large plenary program, and I also like the idea of including talks on top papers/developments from other theory conferences or related fields. On the other hand, I do share the concerns some have raised that this will shift the spotlight of the meeting more onto prominent senior people. One thing that I find very nice about our field (as compared to many other sciences) is that even a complete unknown student can submit a strong result to a conference and get just as much attention from the community as someone well-established (unless these two are scheduled in parallel… another reason for plenary STOC talks) A mainly “invited” plenary session can shift that balance in a way that might be very unhealthy for the field. Things that can help address this are (a) having the STOC PC rather than a separate organizing committee select the plenary STOC papers (indeed, I’d prefer that the entire STOC program be plenary per my colocation comment above), and (b)to the extent possible, invite papers rather than people. (Yes, we might get some bad plenary talks as a result… but this is also how we train our students to speak to broad audiences rather than just to specialists.)
– 5 min talks and posters. While I personally think a 5 min talk and a poster can be an effective way to present a paper, I do have some hesitation about accepting STOC papers into this format. When someone submits, they should have an idea what kind of platform they are going to get if accepted, as their very decision to submit may depend on this. I think the PC choosing whether to give them a parallel vs. a plenary talk is OK, but 5 min+poster is too different from the others. I’d be inclined to keep the poster session wide open and have it not come with a publication that constrains submissions to other conferences. There is no need to put a significant filter on the poster session, unless it starts to get too crowded after a few years. One could also create a “rump session” of 5 min talks where people can pitch their posters.
All of these general principles are great, except you may want to do some calculations and come up with a specific plan. (Apart from the colocation, which is not under our control, we have been driven by the same principles.)
The Sigact EC had some of the same reservations as you did, and it has been enlightening to see them wrestle with the overall constraint satisfaction problem, and conclude that some tradeoffs need to be made. I encourage you to do the same (I will send you a concrete plan by email.)
The 5-min talk proposal should be seen as *adding* rather than subtracting. It allows about 55+ 20min talks in <= 2 parallel sessions, which is within spitting distance of total number of 70 accepts in some recent STOC/FOCS. By allowing the 5 min talk you get the ability to accept 60+ more papers; we have to decide whether those papers are simply rejected, or whether they get some kind of limited exposure.
Thanks, Sanjeev, I will do some calculations on my flight to Portland tomorrow. 🙂
Re the 5-min talk proposal, I do like it as additive to the program, but I fear that it will not be additive from the author’s perspective. (I know that we want to focus on what’s good for the audience, but we need to align the authors’ incentives so that people actually come and present the work we want to hear.) Personally, I would rather have a full talk slot at CCC, TCC, etc. than have a 5-min talk+poster at STOC (even if I get full STOC publication on my CV). To make it additive from the author’s perspective, the 5 min talk+poster should not preclude a full talk at another conference, which means it should not come with a full published paper in the proceedings – in such a case only minimal screening should be needed and it may as well be done separately from the regular PC process. But maybe my own preferences are different than most authors in the community!
>> – not dramatically changing the current STOC paper acceptance+publication process, so as not to break something that is working reasonably well (albeit not perfectly)
In what sense is it working reasonably well? it single handedly did away with the main role and mandate of STOC/FOCS as flagship conference for best TCS papers.
This is once more people inside the restaurant comparing notes with each other and announcing to the rest of the world that there is nothing wrong with menu, just as the place slowly withers away for lack of business.
I agree with Alex that things are not well with STOC/FOCS, very far from well.
My own critique and suggestions were articulated in a prior post in this series, and can be found in http://www.wisdom.weizmann.ac.il/~oded/on-s17.html
In my opinion, what is wrong about the current STOC/FOCS is
* too little interest in the actual contents of the venue; and
* too much preoccupation with the effect of the venue on one’s CV.
(See details in the aforementioned webpage.)
Let me reproduce my operational suggestion: I suggest that the PC uses a “soft decision” rather than a hard threshold. Rather than deciding whether or not to accept a submission, the PC will decide whether to accept it, how much time to allocate it, and at what forum. (Since the authors will be notified of this practice a priori, I see no problem wrt their expectations.) These decisions need not be based solely on the quality of the work; they may be based on an estimate of who may gain from hearing about the work and how much time is most cost-effective for the communication of the work’s contents.
Options may include:
* an X-minute presentation in a plenary session, where X is, say, in {30, 20, 10, 5};
* an X-minute presentation in one of Y parallel specialized sessions, where X is as above and Y may be in {2, 3, 4, 5}.
There is no clear ranking between many of these 4 times 5 configurations; the PC should just select what it thinks bests suits the submission.
Oded:
I can see that the kind of conference you envisage would be more fun to attend. I enjoyed Boaz’s decision as PC chair to move some fraction of talks into plenary sessions at a recent FOCS; this gave it a curated feel.
What I don’t see is why you think your plan would decouple the preoccupation with CV brownie points. These decisions (plenary talk vs talk in N parallel sessions) will also be scrutinized and reported on CVs.
Perhaps what you mean to say is that we can’t do anything about the CV preoccupation, so we might as well forget about it and have more interesting conferences?
Sanjeev
ps Just to clarify, our committee has no desire or mandate to change the PC process, except our planning needs to know how much time to allocate to STOC sessions in the festival.
I certainly agree that there is much to be improved about STOC, but this has already been discussed at length, and my intention was mainly to agree with the task force’s mission and focus on the concrete suggestions for how to improve STOC. Like Oded, my feeling is that the main problem with STOC is that we collectively have not paid enough attention to the value of the event for the attendees (as opposed to the CV’s of the authors), and I am happy that this is where the task force is devoting its attention. I think it is good that they have partly decoupled the “paper publication” aspect of STOC from figuring out how to improve STOC as an event. While there are certainly things to be improved in the former, it seems that the community has very divergent (and strong) views on what to do about it and we shouldn’t let such disagreement stop us from improving the latter. It is my hope that by refocusing attention on the event and the audience, this process will also have a positive effect on how PCs think about accepting and rejecting papers to STOC.
To Oded’s specific proposal, I see merits in a continuum of talk possibilities, though I worry about making the PC’s already difficult task even more complicated (as this involves the PC deciding on many more bits of information). I am supportive of the idea of having at least two different types of “acceptances” – plenary talk and parallel talk. My worry about going as extreme as 5min talks is not that it is unfair to authors — we can be completely transparent about this. It is that authors may simply stop submitting. To go to an extreme (though not far from Sanjeev’s proposal), if half of all STOC papers got only 5min talks, I would probably stop submitting papers to STOC. This is not about the effect on my CV (which might still be better with the STOC paper). Rather, it is about doing our job (as scientists) to try as hard as we can to expose our work to those who might be interested in it. My feeling is that a 20-30min talk in a specialized conference would often achieve that much better than a 5min talk at STOC. As I said earlier, if the 5min talk did not come with a publication that precludes me from submitting to another conference, then that completely changes things…
ps an easy solution to my concern with 5min talks is to allow authors accepted for 5min talks to opt out of the proceedings (and hence free themselves to submit to another conference). in such a case, I would have no objection to them at all, and actually would support them as a way to give the PC more freedom in forming a program that is valuable for attendees.
Salil, that is something that the community can allow. The logistical details can be worked out later.
I think that’s actually a good idea. It’s best that authors do not need to make complicated probabilistic reasonings in deciding whether to submit, and giving them this “opt out” option allows them not to worry about this.
The proposal from Oded brings STOC/FOCS much close to where it should be if it wishes to remain relevant.
First a larger maximum number of acceptances in the top tier track (with a number of reserved-but-not-guaranteed slots for areas of presently little coverage) so that the program is large enough to capture the best work in all areas of TCS.
Second, a catch process, optional to the authors, a la INFOCOM A/INFOCOM B. In this second track excellent but-not-quite-as-outstanding papers are given the choice to appear in an alternate track of distinction. There are at least three reasons why this makes sense: (1) it brings in more presenters/audience (2) it enlarges the coverage of the event as a whole without lowering the quality standards of the main track (3) it reduces reviewer load on the entire system.
The PC/talk-scheduling committee should be aiming for:
~15 plenary talks of 40 minutes or so each single session
~90 talks of 20 minutes each in parallel sessions
~90 talks of 5 minutes each+poster sessions also in parallel sessions
That is an event worth attending: larger number of talks at top level, large number of high quality papers in and outside one’s area worth looking into and lastly a large number of talks of more limited interest of which one would attend those in one’s own area of research.
TL;DR for the entire discussion: more papers in more areas with more parallel sessions all while maintaining high quality.