STOC 2016 just ended and it included many great results with one highlight of course being Laci Babai’s quasipolynomial time algorithm for graph isomorphism. But today I wanted to mention another paper that I found quite interesting and reminded me of the famous Tolstoy quote that
Happy families are all alike; every unhappy family is unhappy in its own way.
I am talking about the work Reed-Muller Codes Achieve Capacity on Erasure Channels by Shrinivas Kudekar, Santhosh Kumar, Marco Mondelli, Henry D. Pfister, Eren Sasoglu and Rudiger Urbanke. We are used to thinking of some error correcting codes as being “better” than others in the sense that they have fewer decoding errors. But it turns out that in some sense all codes of a given rate have the same average number of errors. The only difference is that “bad” codes (such as the repetition code), have a fairly “smooth” error profile in the sense that the probability of decoding success decays essentially like a low degree polynomial with the fraction of errors, while for “good” codes the decay is like a step function, where one can succeed with probability when the error is smaller than some but this probability quickly decays to half when the error passes .
Specifically, if is a linear code of dimension and , we let be the random variable over that is obtained by sampling a random codeword in and erasing (i.e., replacing it with ) every coordinate independently with probability . Then we define to be the average over all of the conditional entropy of given . Note that for linear codes, the coordinate is either completely fixed by or it is a completely uniform bit, and hence can be thought of as the expected number of the coordinates that we won’t be able to decode with probability better than half from a -sized random subset of the remaining coordinates.
One formalization of this notion that all codes have the same average number of errors is known as the Area Law for EXIT functions which states that for every code of dimension , the integral is a fixed constant independent of . In particular note that if is the simple “repetition code” where we simply repeat every symbol times, then the probability we can’t decode some coordinate from the remaining ones (in which case the entropy is one) is exactly where is the erasure probability. Hence in this case we can easily compute the integral which is simply one minus the rate of the code. In particular this tells us that the average entropy is always equal to the rate of the code. A code is said to be capacity achieving if there is some function that goes to zero with such that whenever . The area law immediately implies that in this case it must be that is close to one when (since otherwise the total integral would be smaller than ), and hence a code is capacity achieving if and only if the function has a threshold behavior. (See figure below).
The paper above uses this observation to show that the Reed Muller code is capacity achieving for this binary erasure channel. The only property they use is the symmetry of this code which means that for this code we might as well have defined with some fixed coordinate (e.g., the first one). In this case, using linearity, we can see that for every erasure pattern on the coordinates the entropy of given is a Boolean monotone function of . (Booleanity follows because in a linear subspace the entropy of the remaining coordinate is either zero or one; monotonicity follows because in the erasure channel erasing more coordinates cannot help you decode.) One can then use the papers of Friedgut or Friedgut-Kalai to establish such a property. (The Reed-Muller code has an additional stronger property of double transitivity which allows to deduce that one can decode not just most coordinates but all coordinates with high probability when the fraction of errors is a smaller than the capacity.)
How do you prove this area law? The idea is simple. Because of linearity, we can think of the following setting: suppose we have the all zero codeword and we permute its coordinates randomly and reveal the first of them. Then the probability that the coordinate is determined to be zero as well is . Another way to say it is that if we permute the columns of the generating matrix of randomly, then the probability that the column is independent from the first columns is . In other words, if we keep track of the rank of the first columns, then at step the probability that the rank will increase by one is , but since we know that the rank of all columns is , it follows that , which is what we wanted to prove. QED
p.s. Thanks to Yuval Wigderson, whose senior thesis is a good source for these questions.
On October 5-8 (right before FOCS 2016) there will be a workshop in Princeton in honor of Avi Wigderson’s 60th birthday. Avi is one of the most productive, generous and collaborative researchers in our community (see mosiac below of his collaborators). So, it’s not surprising that we were able to get a great lineup of speakers to what promises to be a fantastic workshop on Computer Science, Mathematics, and their interactions.
Attendance is free but registration is required. Also there are funds for travel support for students for which you should apply before August 1st.
Confirmed speakers are:
- Scott Aaronson – MIT
- Dorit Aharonov – Hebrew University of Jerusalem
- Noga Alon – Tel Aviv University
- Zeev Dvir – Princeton University
- Oded Goldreich – Weizmann Institute of Science
- Shafi Goldwasser – MIT
Weizmann Institute of Science
- Russell Impagliazzo –
University of California, San Diego
- Gil Kalai – Hebrew University of Jerusalem
- Richard Karp – University of California, Berkeley
- Nati Linial – Hebrew University of Jerusalem
- Richard Lipton – Georgia Institute of Technology
- László Lovász – Eötvös Loránd University
- Alex Lubotzky – Hebrew University of Jerusalem
- Silvio Micali – MIT
- Noam Nisan – Hebrew University of Jerusalem
- Toniann Pitassi – University of Toronto
- Alexander Razborov – University of Chicago
- Omer Reingold – Samsung Research America
- Michael Saks – Rutgers University
- Ronen Shaltiel – University of Haifa
- Madhu Sudan – Harvard University
- Eyal Wigderson – Hebrew University of Jerusalem
Today we witnessed an exciting moment for privacy, CS theory, and many friends and contributors of this blog. The definition of differential privacy, first articulated in a TCC paper just 10 short years ago, became a top-level feature of iOS, announced today at the Apple keynote address. Check this out for yourself:
You may be intrigued as I am by the bold claim by Prof. Aaron Roth, who said that
Incorporating differential privacy broadly into Apple’s technology is visionary, and positions Apple as the clear privacy leader among technology companies today.
Learning more about the underlying technology would benefit the research community and assure the public of validity of these statements. (We, at Research at Google, are trying to adhere to the highest standards of transparency by releasing Chrome’s front-end and back-end for differentially private telemetry.)
I am confident this moment will come. For now, our heartfelt congratulations to everyone, inside and outside Apple, whose work made today’s announcement possible!
By Boaz Barak and Omer Reingold
Yesterday Hillary Clinton became the first woman to be (presumptively) nominated for president by a major party. But in the eyes of many, the Republican Party was first to make history this election season by breaking the “qualifications ceiling” (or perhaps floor) in their own (presumptive) nomination.
Though already predicted in 2000 by the Simpsons , the possibility of a Trump presidency has rattled enough people so that even mostly technical bloggers such as Terry Tao and Scott Aaronson felt compelled to voice their opinion.
We too have been itching for a while to weigh in and share our opinions and to use every tool in our disposal for that, including this blog. We certainly think it’s very appropriate for scientists to be involved citizens and speak up about their views. But though we debated it, we felt that this being a group (technical) blog, it’s best not to wage into politics (as long as it doesn’t directly touch on issues related to computer science such as the Apple vs. FBI case). Hence we will refrain from future postings about the presidential election. For full disclosure, both of us personally support Hillary Clinton and have been donating to her campaign.
The last few weeks have seen amazing results in additive combinatorics, where following a breakthrough by Croot, Lev and Pach, several longstanding open questions have been resolved using short simple proofs. I haven’t been following this progress, but fortunately Bobby Kleinberg gave an excellent talk yesterday in our reading group about some of these works, and their relations to TCS questions such as approaches for fast matrix multiplication. Since the proofs are short and these results also been extensively blogged about, what follows is mainly for my own benefit.
Among other things, Bobby showed the proof of the following result, that demonstrates much of those ideas:
Theorem: (Ellenberg and Gijswijt, building on Croot-Lev-Pach) There exists an absolute constant such that for every , if then contains a 3-term arithmetic progression.
To put this in perspective, up till a few weeks ago, the best bounds were of the form and were shown using fairly complicated proofs, and it was very reasonable to assume that a bound of the form is the best we can do. Indeed, an old construction of Behrend shows that this is the case in other groups such as the integers modulo some large or where is some large value depending on . The proof generalizes to for every constant prime (and for composite order cyclic groups as well).
The proof is extremely simple. It seems to me that it can be summarized to two observations:
- Due to the algebraic structure of the problem, one can “interpolate” in some sense a -a.p. free set with a polynomial of degree that is a half as small than you would expect otherwise.
- Due to concentration of measure phenomena in finite fields, this constant multiplicative factor makes a huge difference. There are values for which polynomials of degree make up all but an exponentially small fraction of the functions from to , while polynomials of degree only constitute an exponentially small fraction of these functions.
Let’s now show the proof. Assume towards a contradiction that satisfies ( can be some sufficiently small constant, will do) but there do not exist three distinct points that form a -a.p. (i.e., such that or, equivalently, ).
Let be the number of -variate monomials over where each variable has individual degree at most (higher degree can be ignored modulo ) and the total degree is at most . Note that there are possible monomials where each degree is at most two, and their degree ranges from to , where by concentration of measure most of them have degree roughly . Indeed, using the Chernoff bound we can see that if is a sufficiently small constant, we can pick some such that if then but (to get optimal results, one sets to be roughly and derives from this value).
Now, if we choose in that manner, then we can find a polynomial of degree at most that vanishes on but is non zero on at least points. Indeed, finding such a polynomial amounts to solving a set of linear equations in variables.1 Define the matrix such that . Since the assumption that implies that , the theorem follows immediately from the following two claims:
Claim 1: .
Claim 2: .
Claim 1 is fairly immediate. Since is -a.p. free, for every , is not in and hence is zeros on all the off diagonal elements. On the other hand, by the way we chose it, has at least nonzeroes on the diagonal.
For Claim 2, we expand as a polynomial of degree in the two variables and , and write where corresponds to the part of this polynomial where the degree in is at most and corresponds to the part where the degree in is larger and hence the degree in is at most . We claim that both and are at most . Indeed, we can write as for some coefficients and polynomials , where indexes the monomials in of degree at most . But this shows that is a sum of at most rank one matrices and hence . The same reasoning shows that thus completing the proof of Claim 2 and the theorem itself.
More formally, we can argue that the set of degree polynomials that vanish on has dimension at least and hence it contains a polynomial with at least this number of nonzero values.↩
Actually, not really…
Northwestern University held a workshop on semidefinite programming hierarchies and sum of squares. Videos of the talks by Prasad Raghavendra, David Steurer and myself are available from the link above. The content to unicorns ratio in Prasad and David’s talks is much higher ☺
Tomorrow, Wednesday May 25, is the international Towel Day in honor of Douglas Adams, author of the 5-book trilogy “The hitchhiker’s Guide to the Galaxy”. In his book (and his prior 1978 radio series) Adams gave a nice illustration of computational complexity and non uniform computation in his story about the “deep thought” computer who took 7.5 million years to answer the ultimate question of life, the universe, and everything, though today Google’s calculator can do this in 0.66 seconds.
If you want to celebrate towel day in Cambridge, bring your towel to Tselil Schramm’s talk on refuting random constraint satisfaction problems using Sums of Squares in MIT’s algorithms and complexity seminar (4pm, room 32-G575).