Algorithmic and Information Theoretic Decoding Thresholds for Low density Parity-Check Code

by Jeremy Dohmann, Vanessa Wong, Venkat Arun

Abstract

We will discuss error-correcting codes: specifically, low-density parity-check (LDPC) codes. We first describe their construction and information-theoretical decoding thresholds, $p_{c}$ .

Belief propagation (BP) (see Tom’s notes) can be used to decode these. We analyze BP to find the maximum error-rate upto which BP succeeds.

After this point, BP will reach a suboptimal fixed point with high probability. This is lower than the information-theoretic bound, illustrating a gap between algorithmic and information-theoretic thresholds for decoding.

Then we outline a proof of a theorem suggesting that any efficient algorithm, not just BP, will fail after the algorithmic threshold. This is because there is a phase transition at the algorithmic threshold, after which there exist an exponentially large number of suboptimal `metastable’ states near the optimal solution. Local search algorithms will tend to get stuck at these suboptimal points.

Introduction to Error Correcting Codes

Motivation

Alice wants to send a message to Bob, but their channel of communication is such that Bob receives a corrupted version of what Alice sent. Most practical communication devices are imperfect and introduce errors in the messages they are transmitting. For instance, if Alice sends 3V, Bob will really receive three volts plus some noise (we have chosen to ignore some inconvenient practical details here). In many cases, this noise is quite small, e.g. it could be less than 0.5V in 99.99% of cases. So, in principle, Alice could have had very reliable delivery by just choosing to always send 0V for a logical 0 and 5V for logical 1, using checksums to detect the occasional error. But this is wasteful. Alice could have squeezed more levels between 0 and 5V to get a higher bitrate. This causes errors, but Alice can introduce redundancy in the bits she is transmitting which can enable Bob to decode the correct message with high probability. Since it is much easier to control redundancy in encoding than in physical quantities, practical communication devices often choose choose to pack enough bits into their physical signal that errors are relatively quite likely, relying instead on redundancy in their encoding to recover from the errors. Redundancy is also used in storage, where we don’t have the option of detecting an error and retransmitting the message.

Some errors in communication are caused by thermal noise. These are unpredictable and unavoidable, but the errors they cause can be easily modeled; they cause bit-flips in random positions in the message. There are other sources of error. The clocks on the two devices may not be correctly synchronized, causing systematic bit-flips in a somewhat predictable pattern. A sudden surge of current (e.g. because someone turned on the lights, and electricity sparked between the contacts) can corrupt a large contiguous segment of bits. Or the cable could simply be cut in which case no information gets through. These other kinds of errors are often harder to model (and easier to detect/mitigate), so we have to remain content with merely detecting them. Thus for the remainder of this blog post, we shall restrict ourselves to an error model where each bit is corrupted with some fixed (but potentially unknown) probability, independent of the other bits. For simplicity, we shall primarily consider the Binary Erasure Channel (BEC), where a bit either goes through successfully, or the receiver knows that there has been an error (though we will introduce some related channels along the way).

Claude Shannon found that given any channel, there is a bitrate below which it is possible to communicate reliably with vanishing error rate. Reliable communication cannot be achieved above this bitrate. Hence this threshold bitrate is called the channel capacity. He showed that random linear codes are an optimal encoding scheme that achieves channel capacity. We will only briefly discuss random linear codes, but essentially they work by choosing random vectors in the input space and mapping them randomly to vectors in the encoded space. Unfortunately we do not have efficient algorithms for decoding these codes (mostly due to the randomness in their construction), and it is conjectured that one doesn’t exist. Recently Low-Density Parity Check (LDPC) codes have gained in popularity. They are simple to construct, and can be efficiently decoded at error levels quite close to the theoretical limits.

With LDPC codes, there are three limits of interest for any given channel and design bitrate (M/N): 1) the error level upto which an algorithm can efficiently decode them, $\epsilon_d$ , 2) the error level level upto which they can be decoded by a computationally unbounded decoder, $\epsilon_c$ and, 3) the error level beyond which no encoding scheme can achieve reliable communication, $\epsilon_s$ . Obviously, $\epsilon_d \le \epsilon_c \le \epsilon_s$ , and in general these inequalities can be strict. Our goal here is to study the gap between $\epsilon_d$ and $\epsilon_c$ . We will sometimes refer to these three quantities as $p_{d}$ , $p_{c}$ , and $p_{shannon}$ when discussing channels besides the BEC. This is following the notation of Mezard and Montanari (2009).

More formally, information theory concerns reliable communication via an unreliable channel. To mitigate the errors in message transmission, error correcting codes introduce some type of systematic redundancy in the transmitted message. Encoding maps are applied to the information sequence to get the encoded message that is transmitted through the channel. The decoding map, on the other hand, is applied to the noisy channel bit (see Figure below). Each message encoded is comprised of $M$ bits and $N>M$ redundant sequences of bits in an error correcting code. $2^M$ possible codewords form a “codebook” $|\mathbb{C}|$ in binary space $\{0,1\}$ .

Claude Shannon’s code ensembles proved that it is easier to construct stochastic (characterized by good properties and high probability) models vs. deterministic code designs. Stochastic models were able to achieve optimal error correcting code performance in comparison to a more rigidly constructed model, proving that it was possible to communicate with a vanishing error probability as long as the rate of transmission $R=M/N$ is smaller than the channel capacity, a measure of the maximum mutual information between channel input and output.

**Fig 1** Schematic of the communication model of information communication.

Thus, in order to construct an optimal error correcting code, one must first define the subset of the space of encoding maps, endow the set with probability distributions, and subsequently define the associated decoding map for each of the encoding maps in the codes. We have included a section in the A that gives a thorough discussion of random code ensembles which are known to achieve optimal decoding, whether via scoring decoding success by bit error rate or decoded word error rate. We will also show a perspective which uses principles from statistical physics to unify the two (often called finite-temperature decoding). From hereon out we will discuss LDPC and explore how the values of $p_{d}$ , $p_{c}$ and $p_{shannon}$ for various channels reveal deep things about the structure of the decoding solution space.

Low-density Parity Check Code

LDPC codes are linear and theoretically excellent error correcting codes that communicate at a rate close to the Shannon capacity. The LDPC codebook is a linear subspace of $\{0,1\}^N$ . For an MxN sparse matrix $\mathbb{H}$ , the codebook is defined as the kernel:

$\mathbb{C} = { \underline{x} \in \{0,1\}^N:\mathbb{H}\underline{x}=\underline{0}}$

where all the multiplications and sums involved in
$\mathbb{H} \underline{x}$ are computed modulo 2.

Matrix $\mathbb{H}$ is called the parity check matrix and the size of the codebook is $2^{N-rank(\mathbb{H})}$ . Given this code, encoding is a linear operation when mapping an $N x L$ binary generating matrix $\mathbb{G}$ (the codebook is the image of $\mathbb{G}:\mathbb{C}={x=\mathbb{G}\underline{z}, \text{where } \underline{z} \in \{0,1\}^L}$ ) such that $\underline{z}\rightarrow \underline{x} =\mathbb{G}z$ ).

Every coding scheme has three essential properties that determine its utility: the geometry of its codebook and the way it sparsely distributes proper codewords within the encoding space $\{0,1\}^{N}$ (c.f. our mention of sphere packing as a geometric analogy for RLCs), the ease with which one can construct a code which sparsely distributes codes within the encoding space, and the existence of fast algorithms to perform effective decoding.

A coding scheme over a given channel (whether it be BSC, BEC, AWGN, etc.) also has three parameters of interest, $p_{d}$ which is the error rate above which some chosen algorithm cannot perform error-free decoding, $p_{c}$ above which even exhaustively enumerating over all $2^{M}$ codewords in the codebook and calculating the MAP probability does not successfully decode, $p_{shannon}$ which is the capacity of the channel, an error rate above which no decoding scheme could perform error-free decoding.

LDPC codebook geometry and $p_{c}$

On the subject of codebook geometry, it is well known that LDPC ensembles, in expectation, produce sparsely distributed codewords. This means that valid codewords (i.e. those that pass all parity checks) are far apart from one another in Hamming space and thus require a relatively large number of bits to be lost in order for one codeword to degenerate into another one. There is an important property called the distance enumerator which determines the expected number of codewords in a tight neighborhood of any given codeword. If for a given distance the expected number is exponentially small, then the coding scheme is robust up to error rates causing that degree of distortion. We discuss a proof of LDPC distance properties in Appendix B and simply state here that LDPCs are good at sparsely distributing valid codewords within the encoding space. The property $p_{c}$ , introduced above is intimately related to the geometry of the codebook and the properties of the noisy channel being decoded over.

The information theoretic threshold, $p_{c}$ is the noise level above which MAP decoding no longer successfully performs decoding. $p_{c}$ is important because it is the error value above which we could theoretically always perform (slow) decoding below $p_{c}$ by enumerating all $2^{M}$ codewords in the codebook and calculating the one with the highest MAP probability.

Every LDPC ensemble has some different $p_{c}$ for a given channel. WFor now we will simply remark that LDPC ensembles are effective because they can be chosen such that $p_{c}$ closely approaches the $p_{shannon}$ limit for many channels. Even more importantly, we will show that it is likely that there is no fast algorithm for which $p_{d}$ = $p_{c}$ for general LDPCs over a general noisy channel.

We will not derive the $p_{c}$ for any LDPC ensembles over any channels (I recommend chapter 15 of this book for details), but we will, in section 3, present results derived by other researchers.

Ease of construction

On the subject of ease of construction and ease of decoding, there is a much simpler graphical representation of LDPC codes which can be used to demonstrate LDPC tractability.

LDPCs can be thought of as bipartite regular graphs, where there are N variable nodes which are connected to M parity check nodes according to randomly chosen edges based on the degree distribution of the LDPC. Though the appendix discusses general degree distributions we will discuss here only (d,k) regular bipartite graphs, in which all variables have d edges and all parity check nodes have k edges, and how to generate them under the configuration model.

The configuration model can be used to generate a bipartite graph with (d,k) degree distribution by initially assigning all variable nodes d half-edges, all parity check nodes k half-edges, and then randomly linking up half-edges between the two sets, deleting all nodes which end up being paired an even number of times, and collapsing all odd numbered multi-edges into a single edge. This system doesn’t work perfectly but for large N, the configuration model will generate a graph for which most nodes have the proper degree. Thus it is relatively easy to generate random graphs which represent LDPC codes of any desired uniform degree distribution. An example of this graphical representation is in figure 2

**Fig 2** Factor graph of a (2,3) regular LDPC code with factor nodes as black squares and variable nodes as white circles, and notation for BP messages.

How the graphical model relates to fast decoding

The graphical model of LDPCs is useful because it is both easy to construct and presents a natural way to perform fast decoding. In fact, the fast graph-based decoding algorithm, Belief Propagation, we use has a $p_{d}$ which likely represents the upper limit on fast decoding for LDPCs.

We have seen recently that bipartite graphical models
which represent a factorized probability distribution can be used to calculate marginal probabilities of individual variable nodes (what a mouthful!).

Basically, if the structure of the graph reflects some underlying probability distribution (e.g. the probability that noisy bit $y_i$ was originally sent as bit $x_i$ ) then we can use an iterative algorithm called Belief Propagation (see blog post above) to actually converge to the exact probability distribution for each $P(y_i~|~x_{i})$ .

This is important because when we perform decoding, we would like to estimate the marginal probability of each individual variable node (bit in our received vector), and simply set the variable to be the most likely value (this is known as bit-MAP decoding, discussed earlier). As mentioned above, under certain conditions the Belief Propagation algorithm correctly calculates those marginal probabilities for noise rates up to an algorithmic threshold $p_{d}$ .

The Belief Propagation algorithm is an iterative message passing algorithm in which messages are passed between variable nodes and parity check/factor nodes such that, if the messages converge to a fixed point, the messages encode the marginal probabilities of each variable node. Thus BP, if it succeeds can perform bit-MAP decoding and thus successfully decode.

We will show in the next section how the configuration model graphs map to a factorized probability distribution and mention the $p_{d}$ for BP. In the following section we will show an example of decoding over the binary erasure channel, then finally we will show motivation to suggest that the $p_{d}$ for BP over LDPCs represents a hard upper limit above which no fast decoding algorithms exist.

Decoding Errors via Belief Propagation

As mentioned above (again, please see Tom’s excellent blog post for details), the belief propagation algorithm is a useful inference algorithm for stochastic models and sparse graphs derived from computational problems exhibiting thresholding behavior. As discussed, symbol/bit MAP decoding of error correcting codes can be regarded as a statistical inference problem. In this section, we will explore BP decoding to determine the threshold for reliable communication and according optimization for LDPC code ensembles in communication over a binary input output symmetric memoryless channel (BSC or BMS).

Algorithm Overview

Recall that the conditional distribution of the channel input $\underline{x}$ given the output $\underline{y}$ is given by and that we wish to find the $\underline{x}$ that maximizes the below probability given $\underline{y}$

$p(\underline{x}|\underline{y}) = \frac{1}{Z(y)}\prod_{i=1}^{N}Q(y_i|x_i) \prod_{a=1}^{M} \mathbb{I}(x_{i_{1^{a}}} \otimes ... ~x_{k(a)^a} = 0)$ (1)

Where $Q(y_i~|~x_{i})$ is the conditional probability of $y_{i}$ of observing noisy bit $y_{i}$ given that $x_{i}$ was sent. For the BSC we have $Q(y_{i} = 1 | x_{i} = 1) = Q(y_{i} = 0 | x_{i} = 0) = 1-p$ and $Q(y_{i} = 1 | x_{i} = 0) = Q(y_{i} = 0 | x_{i} = 1) = p$ .

Furthermore

$\mathbb{I} (x_{i_{1^{a}}} \otimes ... ~x_{k(a)^a} = 0)$

is an indicator variable which takes value $1$ if $\underline{x}$ satisfies parity check a and 0 otherwise. In particular, the product of these indicators takes into account the fact that 0 probability should be assigned to hypotheses $\underline{x}$ which aren’t in the code book (indicated by at least one parity check failing).

We would like to design a message passing scheme such that the incoming messages for a given variable node encode their marginal probabilities $p(x_i~|~y_i)$ .

Note, first and foremost that this probability can be factorized a la BP factor graphs such that there is a factor node for each parity check node $a$ which contributes probability

$\mathbb{I} (x_{i_{1^{a}}} \otimes ... ~x_{k(a)^a} = 0)$

and a factor node for each channel probability term $Q(y_i~|~x_{i})$ . Since each channel probability term is only connected to a single variable, its message never gets updated during BP and so we omit it from the factor graphs (e.g. note that figure 2 only has parity check nodes and variable nodes)

The message passing scheme ends up taking the form

$v_{i\rightarrow a}^{(t+1)}(x_i) \propto Q(y_{i} | x_{i}) \prod_{b \in \partial i \setminus a} \hat{v}{b \rightarrow i}^{(t)}$ (2)

$\hat{v}_{a \rightarrow i}(x_{i}) \propto \sum_{{x_{j}}} \mathbb{I} (x_{i} \otimes x_{j_{1}} ... x_{j_{k-1}} = 0) \prod_{j \in \partial a \setminus i} v_{j\rightarrow a}^{(t)}(x_j)$ (3)

Where $\partial a$ denotes the neighborhood of factor node $a$ and the sum in (3) is over all possible configurations of the neighbors of $a$ not including $i$ .

Messages are passed along the edges as distributions over binary valued variables described by the log-likelihoods

$h_{i\rightarrow a} = \frac{1}{2}\log \frac{v_{i\rightarrow a(0)}}{v_{i\rightarrow a (1)}}$ (4)

$u_{i\rightarrow a} = \frac{1}{2}\log \frac{\hat{v}{i\rightarrow a(0)}}{\hat{v}{i\rightarrow a (1)}}$ (5)

We also introduce the a priori log likelihood for bit $i$ given the received message $y_i$ :

$B_{i} = \frac{1}{2} log \frac{Q(y_{i} | 0)}{Q(y_{i} | 1)}$

Once we parametrize the messages as log-likelihoods, it turns out we can rewrite our update rules in terms of the parametrized values h and u, making updates much simpler:

$h_{i \rightarrow a}^{(t+1)} = B_i + \sum_{b \in \partial i \setminus a} u_{b \rightarrow i}^{(t)}$ (6)

$u_{b\rightarrow i}^{(t)} = atanh{ \prod_{j \in \partial a \setminus i} tanh(h_{j \rightarrow a}^{(t)})}$ (7)

Given a set of messages, we would perform decoding via the overall log likelihood $h_{i}^{(t+1)} = B_i + \sum_{b \in \partial i} u_{b \rightarrow i}^{(t)}$ . Where $x_{i}$ gets decoded to 0 for $h_{i} > 0$ and 1 for $h_{i} < 0$ .

Typically BP is run until it converges to a set of messages that decode to a word in the codebook, or until a max number of iterations have occurred. Other stopping criteria exist such as the messages between time step t and t+1 being all within some small $\epsilon$ of one another.

It is important to note some properties of BP:

BP always terminates in $d$ steps if the factor graph is a tree of depth $d$
It is not known under what circumstances so called “loopy” BP will converge for non-tree graphs

Because factor graphs of LDPC codes are relatively sparse, they appear “locally tree-like”, a property which is believed to play a crucial role in BP convergence over the factorized probability distribution used in LDPC MAP decoding (eqn 1). As mentioned above BP manages to converge on many sorts of non tree-like graphs given that they have “nice” probability distributions. For example the SK model is known to converge even though the underlying factor graph is a complete graph!

It turns out that BP converges under some noise levels for LDPC decoding, and that the threshold at which it fails to converge, $p_{d}$ , represents a phase transition to a generically different regime in the solution space of the codebook. It’s been noted elsewhere that the BP threshold is often the threshold of fast solving for many cool problems; e.g. k-SAT. This is because it is often thought to generically represent the “best” possible local (ergo “fast”) algorithm for those problems

In appendix C we will show some important properties of BP. The following tables summarize important results for several ensembles and channels. Note how close the information theoretic threshold for LDPCs is to the actual shannon limit $p_{shannon}$ for the channels below.

Table 1: Thresholds for BSC
Various thresholds for BP over LDPC codes in a Binary Symmetric Channel

d	k	$p_{d}$	$p_{c}$	Shannon limit
3	4	.1669	.2101	.2145
3	5	.1138	.1384	.1461
3	6	.084	.101	.11
4	6	.1169	.1726	.174

See Mezard and Montanari, 2009 Chapt 15. for this table

Table 2: Thresholds for BEC
Various thresholds for BP over LDPC codes in a Binary Erasure Channel

d	k	$\epsilon_{d}$	$\epsilon_{c}$	Shannon limit
3	4	.65	.746	.75
3	5	.52	.59	.6
3	6	.429	.4882	.5
4	6	.506	.66566	.6667

See Mezard and Montanari, 2009 Chapt 15. for this table

We will now show exact behavior of the (3,6) LDPC ensemble over the binary erasure channel.

Algorithmic Thresholds for Belief Propagation (BP)

Definitions and notation

Definition 1. In a Binary Erasure Channel (BEC), when the transmitter sends a bit $\in {0, 1}$ , the receiver receives the correct bit with probability $1 - \epsilon$ or an error symbol $*$ with probability $\epsilon$ .

For BECs, the Shannon capacity—the maximum number of data bits that can be transmitted per encoded bit—is given by $1 - \epsilon$ .

Definition 2. An $N$ -bit Error Correcting Code (ECC) is defined by a codebook $\mathcal{C} \subset {0, 1}^N$ . The transmitter encodes information as an element of $\mathcal{C}$ . The receiver receives a corrupted version $y$ of the transmitted codeword. To decode, it picks an $x \in \mathcal{C}$ that is most likely given $y$ and the channel characteristics.

For ease of discourse, we have refrained from defining ECC in full generality.

Definition 3. An $N$ -bit $(\lambda, \rho)$ Low Density Parity Check Code (LDPC) is an ECC with $\mathcal{C} = {x | Hx = 0}$ . Here $H$ is an $M \times N$ matrix and arithmetic is over $\mathbb{Z}_2$ . $H$ is a sparse parity-check matrix. $\lambda(x) = \sum_i\lambda_ix^{i-1}$ and $\rho(x) = \sum_i\rho_ix^{i-1}$ are finite polynomials that characterize $H$ ; $\lambda_i$ is the fraction of columns with $i$ $1$ s and $\rho_i$ is the fraction of rows with $i$ $1$ s. Since these are fractions/probabilities, they must be normalized. Hence $\lambda(1) = \rho(1) = 1$ . $H$ is a random matrix, and therefore has full rank with high probability.

In an LDPC code, $|\mathcal{C}| = 2^{N - M}$ . Hence every $N$ bits contain $N-M$ bits of information, making the rate $1 - M / N$ . Over binary erasure channels (BECs), on receiving $y$ , the decoder must choose an $x$ such that $x_i = y_i \forall i$ such that $y_i \neq *$ , and $Hx = 0$ ( $x_i$ , $y_i$ denote the $i^{th}$ bit of $x$ and $y$ respectively). That is, the bits that were successfully transmitted should be preserved; other bits should be chosen to satisfy the parity check equations. If multiple correct choices of $x$ are possible, then $y$ cannot be unambiguously decoded.

BP/Peeling algorithm

In general, decoding can be computationally hard. But there exists an error rate $\epsilon_d$ , a function of $\lambda$ and $\rho$ , below which belief propagation succeeds in decoding. Let $\epsilon_c$ be the maximum error rate upto which successful decoding is possible (i.e. we can unambiguously determine the transmitted codeword) and $\epsilon_s$ be the Shannon limit, then $\epsilon_d \leq \epsilon_c \leq \epsilon_s$ . In general, these inequalities can be strict, illustrating the gap between what is information theoretically possible, and what is computationally feasible.

Belief propagation (BP) for decoding LDPC-codes is equivalent to a simple peeling algorithm. Let us first describe the factor-graph representation for decoding. This is denoted in figure 3. Variables on the left are the received symbol $\in {0, 1, *}$ . Factor nodes on the right denote the parity-check constraint (rows of $H$ ). The XOR of variables connected to each factor node must be 0.

BP/the peeling algorithm works as follows. For simplicity of exposition, consider that the all zeros code-word has been transmitted. Since this is a linear code, there is no loss of generality. At first, only the $1 - \epsilon$ variables that were successfully transmitted are fully determined. In the next round, the factor nodes that have exactly one undetermined variable can determine that variable using their parity-check constraint.

**Fig 3** An example of a received code-word and corresponding parity-check constraints that is information-theoretically determined (only the all zeros codeword satisfies all constraints), but cannot be decoded by the belief propagation algorithm because no factor node has exactly one unknown variable. Here, only the V0 is correctly received. Constraints F1, F2, F3 and F4 imply that V1=V2=V3=V4=V5. If V0=0, then all the rest must be 0s to satisfy F0 (note, the only other valid codeword is all ones).

BP isn’t perfect

This algorithm is not perfect. Figure 3 is an example of a received codeword which can be unambiguously decoded — only the all zeros codeword satisfies all the constraints— but the BP algorithm fails, because at any point, all factor nodes have more than one unknown variable. It seems that the only way to solve problems like that is to exhaustively understand the implications of the parity-check equations. If this examples seems contrived, that is because it is. Decoding becomes harder as the degree and number of constraints increases; we had to add a lot of constraints to make this example work. Fortunately, if the graph is sparse, BP succeeds. We prove this in the following theorem:

Phase transitions for BP

Theorem 1. A $(\lambda, \rho)$ LDPC code can be decoded by BP as $N \rightarrow \infty$ when the error rate is less than $\epsilon_d$ :

$\epsilon_d = \mathrm{inf}_{z \in (0, 1)}\left[\frac{z}{\lambda(1 - \rho(1 - z))}\right]$

Proof. To prove this, let us analyze the density evolution. For BECs, this is particularly simple as we only need to keep track of the fraction of undetermined variables and factor nodes at timestep $t:~latex z_t$ and $\hat{z}_t$ respectively. As $N \rightarrow \infty$ , these fractions are probabilities. A factor node is determined when all of its variables are determined (note: if all but one is determined, the last one can be immediately determined). The following recursion relations hold:

$z_{t+1} = \epsilon\lambda(\hat{z}_t) \:\:\:\mathrm{and}\:\:\: \hat{z}_t = 1 - \rho(1 - z_t)$ (8)

The first holds because a variable node is undetermined at timestep $t+1$ if it was originally undetermined (which happens with probability $\epsilon$ ) and if it isn’t determined in the last step, which happens with probability say $p$ . Now,

$p = \mathbf{P}(degree=2)\hat{z}_t + \mathbf{P}(degree=3)\hat{z}_t^2 + \cdots = \lambda(\hat{z}_t)$

A similar reasoning holds for the second relation. $1 - z_t$ is the probability that a given neighboring variable node is determined. $\rho(1 - z_t)$ is the probability that at-most one is undetermined, and hence this function node is determined. $1 - \rho(1 - z_t)$ is the probability that this function node is undetermined.

Composing the two relations in equation 8, we get the recursion:

$z_{t+1} = F_{\epsilon}(z) = \epsilon \lambda(1 - \rho(1 - z_t))$

An example of $F(z)$ is shown in Figure 4 for $\lambda(x) = x^2, \rho(x) = x^5$ . That is a (3,6) regular graph where variable nodes and function nodes have 3 and 6 neighbors respectively. On the left, $F(z)$ is always below $z$ . Hence the recursion with $z_0$ starting from the far right will converge to the fixed point $F(z) = z = 0$ . But on the right, $\epsilon$ is large enough that $F(z)$ intersects $z$ at a non-zero point. Hence the recursion will converge to the higher fixed point instead, without ever reaching the `correct’ fixed point. BP therefore gets stuck at a suboptimal solution, though information-theoretically a correct solution exists. This can be interpreted as a glassy state, where many deep local minima are present, and BP will converge to the wrong minimum.

The condition for BP to converge is $F_\epsilon(z) \le z \:\: \forall z \in (0, 1)$ . Hence the threshold error rate, $\epsilon_d$ , below which this condition holds is:

$\epsilon_d = \mathrm{inf}_{z \in (0, 1)}\left[\frac{z}{\lambda(1 - \rho(1 - z))}\right]$

For (3, 6) regular graphs, $\epsilon_d \approx 0.429$ ∎

**Fig 4** The recursion relation for a (3,6) regular graph, where $F_\epsilon(z)$ is plotted against $z$ . The identity function $z$ is also shown (in blue). Here $\epsilon_d \approx0.429$ . The graph above and below show the case the error rate is below and above $\epsilon_d$ respectively.

**Fig 4** The recursion relation for a (3,6) regular graph, where $F_\epsilon(z)$ is plotted against $z$ . The identity function $z$ is also shown (in blue). Here $\epsilon_d \approx0.429$ . The graph above and below show the case the error rate is below and above $\epsilon_d$ respectively.

Another interesting phase transition can be observed. As $\epsilon$ increases, for some values of $(\lambda, \rho)$ , the first intersection of $F(z)$ and $F(z)$ happens at a non-zero point. For others, it starts of at $z=0$ and goes up continuously. In the former case, the decoding error rate jumps discontinuously as $\epsilon$ increases from 0 to a non-zero values. For the latter, it increases continuously.

To see the gap between $\epsilon_d$ and what can be information theoretically, we look at what happens when the degrees of the LDPC code is increased while keeping the rate constant. Specifically consider the $(l, k)$ regular graph (i.e. $\lambda(x) = x^{l-1}$ and $\rho(x) = x^{k-1}$ ) as $k \rightarrow \infty$ while $l / k = 0.5$ is fixed. Note that the rate of the code is $1 - l / k$ . This is shown in Figure 5. $\epsilon_d$ decreases toward 0. But as $k \rightarrow \infty$ , it should become information-theoretically easier to decode. In fact, as $k \rightarrow \infty$ , the code approaches a random linear code, which is known to achieve Shannon capacity. Hence we can believe that the information-theoretically achievable decoding rate is non-decreasing. Thus there is a gap between what is information theoretically possible to decode, and what is computationally feasible using Belief Propagation.

**Fig 5** $\epsilon_d$ decreases as $k$ increases, while the rate $= 1 - l/k = 0.5$ is fixed. In fact $\mathrm{lim}_{k \rightarrow \infty}\epsilon_d = 0$ .

Finally we would like to mention that it is possible to choose a sequence of polynomials $(\lambda, \rho)$ such that $\epsilon_d$ approaches the Shannon limit. While it is non-trivial to sample exactly from this distribution, good approximations exist and LDPC codes can achieve close to channel capacity over binary erasure channels.

The solution space in $p_{d}\leq p \leq p_{c}$

The energy landscape of LDPC decoding

We have already shown the exact location of the $p_{MAP} = p_{c}$ threshold above which decoding is not possible for the LDPC ensemble and have also investigated the point at which the BP algorithm fails, $p_{d}$ .

It should not be surprising to us that any given algorithm we attempt to throw at the problem fails at a certain point below $p_{c}$ . In fact there are many simple, random algorithms from the class of Markov-chain Monte Carlos which give fast run times but which fail at values far below even $p_{d}$ . The failing point of a particular algorithm, per se, is not necessarily very significant. We shouldn’t expect that any given algorithm, besides explicitly calculating the symbol MAP by traversing the entire codebook, would be able to achieve $p_{c}$ .

What is of interest to us here, is that $p_{d}$ marks a provable threshold in the solution space of LDPC decoding during which it is likely no locally-based methods, and therefore no fast algorithms can decode with nonzero probability. We will show later precisely the number and energy levels of these metastable states for the BEC. Proof of this transition for other channel types is outside the scope of this lecture.

In this section we will rephrase decoding as an energy minimization problem and use three techniques to explore the existence of metastable states and their effect on local search algorithms.

In particular we will first use a generic local search algorithm that attempts to approximately solve energy minimization expression of decoding.

We will next use a more sophisticated Markov chain Monte Carlo method called simulated annealing. Simulated annealing is useful because it offers a perspective that more closely models real physical processes and that has the property that its convergence behavior closely mimics the structure of the metastable configurations.

Energy minimization problem

To begin, we will reframe our problem in terms of constraint satisfaction.

The codewords of an LDPC code are solutions of a CSP. The variables are the bits of the word and the constraints are the parity check equations. Though this means our constraints are a system of linear equations, our problem here is made more complicated by the fact that we are searching for not just ANY solution to the system but for a particular solution, namely the transmitted codeword.

The received message $\underbar{\textit{y}}$ tells us where we should look for the solution.

Assume we are using the binary-input, memoryless, output-symmetric channel with transition probability $\mathbf{Q}(y | x)$ .

The probability that $\underbar{\textit{x}}$ was the transmitted codeword, given $\underbar{\textit{y}}$ is $\mathbb{P}(\underbar{\textit{x}} | \underbar{\textit{y}}) = \mu_{y}(\underbar{\textit{x}})$

Where

$\mu_{y}(\underline{x}|\underline{y}) = \frac{1}{Z(y)}\prod_{i=1}^NQ(y_i|x_i)\prod_{a=1}^M\mathbb{I}(x_{i_1^a}\otimes ... \otimes x_{k(a)^a} = 0)$ (10)

We can associate an optimization problem with this code. In particular, define $E(\underbar{\textit{x}})$ to be twice the number of parity check equations violated by $\underbar{\textit{y}}$ .

We have already discussed how symbol MAP computes the marginals of the distribution $\mu_{y}(\underbar{\textit{x}})$ and how word MAP finds its argmax.

We shall here discuss two related problems

optimizing the energy function within a subset of the configuration space defined by the received word
sampling from a ’tilted’ Boltzmann distribution associated with the energy

Define the log-likelihood of x being the input given the received y to be

$L_{\underline{y}}(\underline{x}) = \sum_{i=1}^N Q(y_i|x_i)$ (11)

If we assume WLOG that the all zero codeword was transmitted, by the law of large numbers, for large N the log-likelihood $L_{\underbar{\textit{y}}}(\underbar{\textit{x}})$ of this codeword is close to $-Nh$ where $h$ is the channel entropy. The probability of an order-N deviation away from this value is exponentially small.

This suggests that we should look for the transmitted codeword amongst those $\underbar{\textit{x}} \in \mathbb{C}$ such that $L_{\underbar{\textit{y}}}(\underbar{\textit{x}})$ is close to h.

The constraint version of our decoding strategy – known as typical-pairs decoding – is thus, find $\underbar{\textit{x}}$ such that $L_{\underbar{\textit{y}}}(\underbar{\textit{x}}) > -N(h+\delta)$ . This constraint will be referred to as the `distance constraint’ and we should consider the situation where if exactly one codeword satisfies the distance constraint, return it.

Since codewords are global energy minima ( $E(\underbar{\textit{x}}) = 0$ for all $\underbar{\textit{x}} \in \mathbb{C}$ ), we can phrase typical-pairs decoding as an optimization problem

Minimize $E(\underbar{\textit{x}})$ subject to $L_{\underbar{\textit{y}}}(\underbar{\textit{x}}) > -N(h+\delta)$ .

This decoding succeeds iff the minimum is non-degenerate. This happens with high probability for $p < p_{c}$ and with zero probability for $p > p_{c}$ . In particular, there are exponentially many degenerate energy minima above $p_{c}$ .

Similar to what we have seen elsewhere in the course, there exists a generically intermediate regime $p_{d}\leq p \leq p_{c}$ in which the global energy minimum is still the correct codeword bu there is an exponentially large number of local energy minima obscuring it (see figure 6).

What is so special about BP is that the threshold at which these exponentially many metastable states proliferate is exactly the algorithmic threshold $p_{d}$ for BP which we proved earlier.

**Fig 6** A cartoon landscape for the Energy function defined above (number of violated checks). *Left*: The energy has a unique global minimum with $E(\underline{x}) = 0$ (the transmitted codeword) and no local minima. *Center*: many deep local minima appear, although the global minimum is non degenerate. *Right*: more than one codeword is compatible with the likelihood constraint, the global minimum $E(\underline{x}) = 0$ is degenerate *adapted from Mezard and Montanari, 2009 Chapt 21*

**Fig 6** A cartoon landscape for the Energy function defined above (number of violated checks). *Left*: The energy has a unique global minimum with $E(\underline{x}) = 0$ (the transmitted codeword) and no local minima. *Center*: many deep local minima appear, although the global minimum is non degenerate. *Right*: more than one codeword is compatible with the likelihood constraint, the global minimum $E(\underline{x}) = 0$ is degenerate *adapted from Mezard and Montanari, 2009 Chapt 21*

While finding solutions $E(\underbar{\textit{x}}) = 0$ amounts to Gaussian elimination, the constraint $L_{\underbar{\textit{y}}}(\underbar{\textit{x}}) > -N(h+\delta)$ is not a linear constraint. Thus one needs to use some sort of more advanced search procedure to find satisfying vectors $\underline{x}$ .

We will show that if one resorts to local-search-based algorithms, the metastable states above $p_{d}$ block the algorithm. Furthermore, we suggest that the behavior of the local algorithms discussed below are typical of all local search algorithms (including BP) and that it is very likely the case that no fast algorithm exists capable of finding global energy minima without getting caught in the metastable states which proliferate above $p_{d}$ .

Below is the simplest of local search algorithms, $\Delta$ -local search.

**Fig 7** *Excerpted from Mezard and Montanari, 2009 Chapt 21*

Delta search typefies local search algorithms. It walks semi-randomly through the landscape searching for low energy configurations. Its parameter is defined such that, when stuck in a metastable state it can climb out of it in polynomial time if the steepness of its energy barrier is $\leq \Delta$ . Thus its failure in the $p \geq p_{d}$ region suggests that there are no barriers of constant size and that barriers of order N are the norm.

MCMC and the relaxation time of a random walk

We can understand the geometry of the metastable states in greater detail by reframing our MAP problem as follows:

$\mu_{y,\beta}(\underline{x}) = \frac{1}{Z(\beta)}exp{-\beta \cdot E(\underline{x})}\prod_{i=1}^N Q(y_i|x_i)$ (12)

This form is referred to as the `tilted’ Boltzmann because it is a Boltzmann distribution biased by the likelihood function.

In the low temperature limit this reduces to eqn 10 because it finds support only over words in the codebook.

This distribution more closely mimics physical systems. For nonzero temperature it allows support over vectors which are not actually in our codebook but still have low distance to our received message and have low energy – this allows us to probe the metastable states which trap our local algorithms. This is referred to as a code with `soft’ parity check constraints as our distribution permits decodings which fail some checks.

We will use the following algorithm excerpted from Mezard and Montanari Chapt 21:

**Fig 8** *Excerpted from Mezard and Montanari, 2009 Chapt 21*

Where a Glauber update consists of scanning through the bits of the current proposed configuration and flipping the value of bit $i$ with probability

$w_{i}(\underbar{\textit{x}}) = \frac{\mu_{y,\beta}(\underline{x}^{(i)})}{\mu_{y,\beta}(\underline{x}^{(i)}) + \mu_{y,\beta}(\underline{x})}$ (13)

Where $\underline{x}$ is the current configuration and $\underline{x}^{(i)}$ is the configuration obtained by flipping $\underline{x}$ ‘s $i^{th}$ bit

This method is a spin-off of traditional Markov chain Monte-Carlo algorithms with the variation that we lower the temperature according to an annealing schedule that initially assigns probability to all states proportional to the likelihood component of equation 12, allowing the chain to randomly sample the configuration space in the neighborhood of the received noisy word, until in the low temperature limit it becomes concentrated near to configurations which are proper codewords.

This method is useful to us because MCMCs are good models of how randomized methods of local searching for optimal configurations occurs in physical systems. Furthermore, the convergence of MCMCs and the time it takes them to converge tells us both the properties of the energy wells they terminate in and the barriers between minima in the energy landscape.

Let’s now show a property relating convergence times of MCMCs and energy barriers known as the Arrhenius law.

If we take the example of using a simple MCMC random walk with the update rule below over the following landscape

$w(x\rightarrow x') = min \{e^{-\beta [E(x')-E(x)]},~1\}$

**Fig 9** This represents a random walk along a line in which there are two ground states separated by an energy barrier of height $\Delta E$ . *Excerpted from Mezard and Montanari, 2009 Chapt 13*

We find that the expected number of time steps to cross from one well to another is governed by the Arrhenius law $\tau \approx exp{\beta \Delta E}$ .

In general, if there exists a largest energy barrier between any two components of the configuration space (also known as the bottleneck) the time it takes to sample both components, also known as the relaxation time of the MCMC is $\tau_{exp} \geq O(e^{\beta \Delta E})$

With this in mind, we can apply our simulated annealing MCMC to LDPC decoding and examine the properties of the bottlenecks, or metastable states, in our configuration landscape.

Exact values of the metastable energy states for the BEC

It is a well-known consequence of the 1RSB cavity method that the number of metastable states of energy $Ne$ grows like $exp(N\Sigma^{e}(e))$ where $\Sigma^{e}(e)$ is known as the energetic complexity function, a function whose form is implied by the 1RSB cavity equations. This computation can be carried out using a method called Survey Propagation which constructs a factor graph of the messages passed in the original BP factor graph and estimates the values of the marginals of the messages via another round of BP (hence the name 1-step RSB).

Neglecting the actual form of the calculations I will show the following approximate results for the BEC.

**Fig 10** Metastable states for random elements of the (3,6) regular ensemble used over the BEC ( $\epsilon_d = .4294$ and $\epsilon_c = .4882$ . *Left*: the complexity function as a function of energy density above $\epsilon_d$ . *Right*: the maximum and minimum energy densities of metastable states as a function of the erasure probability. Note that at $\epsilon_d \leq .45 \leq \epsilon_c$ latex the curve is positive only for non zero metastable energy densities. This indicates exponentially many metastable states. At erasure rates above $\epsilon_c$ there are exponentially many degenerate ground states. *Excerpted from Mezard and Montanari, 2009 Chapt 21*

**Fig 10** Metastable states for random elements of the (3,6) regular ensemble used over the BEC ( $\epsilon_d = .4294$ and $\epsilon_c = .4882$ . *Left*: the complexity function as a function of energy density above $\epsilon_d$ . *Right*: the maximum and minimum energy densities of metastable states as a function of the erasure probability. Note that at $\epsilon_d \leq .45 \leq \epsilon_c$ latex the curve is positive only for non zero metastable energy densities. This indicates exponentially many metastable states. At erasure rates above $\epsilon_c$ there are exponentially many degenerate ground states. *Excerpted from Mezard and Montanari, 2009 Chapt 21*

In the regime $p \geq p_{d}$ there exists a zero-energy word corresponding to the correct solution. On top of this, there exist non-trivial solutions to the 1RSB method yielding a complexity curve positive in the regime ( $e_{c}, e_{d}$ ). The positive complexity means that there are exponentially many such states and their finite energy means they violate a finite fraction of the parity checks, making their energy wells relatively deep.

As the error rate of the channel increases above $p_{c}$ the minimum energy of the metastable state reaches zero continuously. This means at noise levels above $p_{c}$ there are an exponential number of zero-energy states corresponding to configurations which aren’t code words. These codewords are separated by energy barriers $O(N)$ thus making the relaxation-time of local algorithms, by the Arrhenius law $exp(N)$ in this regime.

**Fig 11** Decoding random codes from the (3,6) ensemble over the BEC. In both cases $N = 10^4$ , and the annealing schedule consists of $t_max = 1000$ equidistant temperatures in $T = 1 / \beta \in [0,1]$ . *Left*: indicates convergence to the correct ground state at $\epsilon = .4 < \epsilon_d$ . *Right*: indicates convergence to approximately the energy density of the highest metastable states $e_d$ (as calculated by the complexity function via 1RSB) for $\epsilon = .6 > \epsilon_{c}$ . *Excerpted from Mezard and Montanari, 2009 Chapt 21*

**Fig 11** Decoding random codes from the (3,6) ensemble over the BEC. In both cases $N = 10^4$ , and the annealing schedule consists of $t_max = 1000$ equidistant temperatures in $T = 1 / \beta \in [0,1]$ . *Left*: indicates convergence to the correct ground state at $\epsilon = .4 < \epsilon_d$ . *Right*: indicates convergence to approximately the energy density of the highest metastable states $e_d$ (as calculated by the complexity function via 1RSB) for $\epsilon = .6 > \epsilon_{c}$ . *Excerpted from Mezard and Montanari, 2009 Chapt 21*

Here you can see a rough sketch of convergence of the simulated annealing algorithm. As the temperature decreases in the $p \leq p_{d}$ regime the algorithm converges to a 0 energy ground state. In the figure on the right we can see that simulated annealing converges to the horizontal line here which corresponds to the energy of the highest metastable state $e_{d}$ for the BEC at $p = .6$ .

Thus we see our local search algorithms end up being attracted to the highest energy of the metastable state.

**Fig 12** Decoding random codes as in figure 11. Here we plot the minimum energy density achieved through simulated annealing plotted as a function of the erasure probability of the BEC. The continuous line indicates the highest lying metastable states as calculated from the complexity function via 1RSB.
*Excerpted from Mezard and Montanari, 2009 Chapt 21*

Though there is not necessarily an exact correspondence between the residual energy at T=0 for simulated annealing and the highest metastable state $e_{d}$ we see that across all values of $p$ that at T=0, $e_{ann} \approx e_{d}$ suggesting local search tends to get caught in the deepest metastable energy wells.

This discussion shows that the algorithmic threshold of BP, $p_{d}$ indicates the onset of a truly different regime within the energy landscape of the codebook. Metastable states of $O(N)$ hight proliferate and become exponentially difficult to escape from via local search methods. Thus the failure of BP likely indicates a regime in which no fast algorithms can perform decoding, even though decoding is still theoretically possible when below $p_c$ , e.g. via exhaustive search of the codebook.

Appendix A: Random Code Ensembles

In an RCE, encoding maps applied to the information sequence are chosen with uniform probability over a solution space. Two decoding schemes are often used and applied to the noise – word MAP and symbol MAP decoding. MAP, otherwise known as “maximum a priori probability” works by maximizing the probability distribution to output the most probable transmission. Word MAP decoding schemes output the codeword with the highest probability by minimizing the block error probability, which is otherwise known as the probability with respect to the channel distribution that the decoded word is different than the true transmitted word. Symbol MAP decoding, on the other hand, minimizes the fraction of incorrect bits averaged over the transmitted code word (bit error rate).

RCE code is defined by the codebook in Hamming space, or the set of all binary strings of length $N$ . In a Hamming space characterized by uniform probability, the number of codewords at a given Hamming distance are a function of the distance enumerator. Distance enumerators take as parameters different weights, given that probabilities of codewords are independent of each other. The distance enumerator shows that, for small enough fractional distance from the true message $x_0$ , the growth rate is negative and the average number of codewords at small distance from $x_0$ vanishes exponentially with N. The Gilbert-Varshamov distance, a lower bound threshold, shows that the the average number of codewords is exponentially large at points where the weight numerator is concentrated.

We look at the performance of RCE code in communication over the Binary Symmetric Channel (BSC), where it is assumed that there is a probability p that transmitted bits will be “flipped” (i.e. with probability p, 1 becomes 0 and 0 becomes 1). With BSCs, channel input and output are the same length N sequences of bits. At larger noise levels, there are an exponential number of codewords closer to $y$ and decoding is unsuccessful. However, decoding via the symbol MAP decoding scheme shows that the $i$ -th bit is decoded by maximizing the marginal probability and amasses contributions from all codewords in the set. Above a threshold, the bit error rate is the same as if the message was transmitted without encoding and decoding, but below this, the RCE seems to work quite well in transmission.

Finite temperature decoding has also been looked at as an interpolation between the two MAP decoding schemes. At low noise, a completely ordered phase can be observed as compared to a glassy phase at higher noise channels. Similar to the a statistical physics model, we can also note an entropy dominated paramagnetic phase at higher temperatures.

Each decoding scheme can be analogized to “sphere packing”, where each probability in the Hamming space distribution represents a sphere of radius r. Decoding schemes have partitions in the Hamming space, so these spheres must be disjoint. If not, intersecting spheres must be eliminated. The lower bound of the remaining spheres is then given by Gilbert-Varshamov bound, whereas the upper bound is dictated by the Hamming distance.

Another random code beside the RCE is the RLC, or random linear code. Encoding in an RLC forms a scheme similar to a linear map, of which all points are equiprobable. The code is specified by an $N x M$ binary matrix, otherwise known as the generating matrix, and it is projected to be error-free below the Shannon capacity.

There are several sources of randomness in codes. Codes are chosen randomly from an ensemble and the codeword to be transmitted is chosen with uniform probability from the code, according to the theorem of source-channel separation. The channel output is then distributed according to a probabilistic process accounting for channel noise and decoding is done by constructing another probability distribution over possible channel inputs and by estimating its signal bit marginal. The decision on the i-th bit is dependent on the distribution. Thus, complications may arise in distinguishing between the two levels of randomness: code, channel input, and noise (“quenched” disorder) versus Boltzmann probability distributions.

Appendix B: Weight enumerators and code performance

The geometric properties of the LDPC codebooks is given by studying the distance enumerator $N_{\underline{x}0}(d)$ to give the number of codewords at Hamming distance d from $\underline{x}_0$ . This takes all-zeros codewords as the reference and uses the weight enumerator, $\mathbb{N}(w)=\mathbb{N}{\underline{x_0}}(d=w)$ as the denomination (number of codewords having weight equal to w). To estimate the expected weight enumerator $\mathcal{N}(w)=\mathbb{E}\mathbb{N}(w)$ for a random code in the $LDPC_N(\Lambda,P)$ ensemble, we know that $\mathbb{N}(w)$ grows exponentially in block-length N, and that each codeword has a weight $w=Nw$ that grows linearly with N. The exponential growth rate $\phi(w)$ is defined by

$\mathcal{N}(w=Nw) = e^{N \phi (w)}$ (14)

denoting an ‘annealed average’, or a disordered system that could be dominated by rare instances in the ensemble. This gives an upper bound on the number of ‘colored factor graphs’ that have an even number of weighted incident edges divided by the total number of factor graphs in the ensemble.

On the other hand, for graphs of fixed degrees with N variable nodes of degree l and M function nodes of degree k, the total number of edges F is given by $F=Mk=Nl$ . A valid colored graph would have $E=wl$ edges, with the number of variable nodes given in ${N\choose w}$ ways, l assignments of weighted sockets to nodes, and l assignments of unweighted sockets to nodes outside the set. If we take $m_r$ to denote the number of function nodes with weighted sockets under the constraints of $\Sigma_{r=0}^km_r=M$ and $\Sigma_{r=0}^krm_r=lw$ , we find the number of ways to color the function node sockets by

$\mathbb{C}(k,M,w) = \sum_{m_{0},...m_{k}}^{even}{M\choose m_{0},...,m_{k}}\prod_{r}{k\choose r}^{m_{r}}$ (15)

$\mathbb{I}\Big(\sum_{r=0}^km_r=M\Big)\mathbb{I}\Big(\sum_{r=0}^krm_r=lw\Big)$ (16)

If we aim to join variable and check nodes so that colorings are matched, knowing that there are $(lw)!(F-lw)!$ possible matchings in each ensemble element, this yields the following formula:

$\mathcal{N}(w)=\frac{(lw)!(F-lw)!}{F!}{N\choose w}\mathbb{C}(k,M,w)$ (17)

At low noise limits, code performance depends on the existence of codewords at distances close to the transmitted codeword. Starting with degree 1 and knowing that the parametric representation for weights is given by

$w = \sum_{l=1}^{l_{max}}\Lambda_l\frac{xy^l}{1+xy^l}$ (18)

derive that

$\phi(w) = -\frac{1}{2}w\log(w/\Lambda_1^2)+O(w)$ (19)

when $x,y,z$ scale to $\sqrt{w}$ . This shows that the exponential growth rate $\phi(w)$ is strictly positive when $w$ is sufficiently small, and that the expected number of codewords within a small Hamming distance from a given codeword is exponential in N. If we take the logarithm of the expected weight enumerator and plot this versus the reduced weight $w=w/N$ for an irregular code with $l_{min}=1$ , we see that $\phi(w)$ is positive near the origin, but that its dervative diverges as $w\rightarrow 0$ . Since this means that each codeword is surrounded by a large number of very close other codewords, this makes the code a very bad ECC and thus, makes it hard to discriminate between codewords at Hamming distances $O(1)$ with noisy observations. Applying this same logic to $l$ of min 2, we still observe that $\phi(w)$ tends to 0 more quickly as $w\rightarrow 0$ in the present case. If we assume that this holds beyond the asymptotic regime, we get

$\bar{\mathcal{N}}(w) = e^{Aw}$ (20)

or that the number of codewords around a particular codeword is $o(N)$ until a Hamming distance $d_* \simeq \log N/A$ , otherwise known as the “effective minimum distance”. For $l_{min} \geq 3$ , we find:

$\phi(w) \simeq \Big(\frac{l_{min}-2}{2}\Big)w\log\Big(\frac{w}{\Lambda_{l_min}}\Big)$ (21)

suggesting that LDPC codes with this property have good short distance behavior. Thus, any error that changes a fraction of the bits smaller than $w_{*}/2$ can be corrected in the absence of codewords within an extensive distance $Nw_{*}$ .

Let us now focus on the capacity of LDPC codes to correct typical errors in a probabilistic channel. For binary symmetric channels that flip each transmitted bit independently with probability $p<\frac{1}{2}$ . If the all-zero codeword $\underline{x}^{(0)} =\underline{0}$ has been transmitted as $\underline{y}$ , whose components are iid random variables that take value 0 with probability $1-p$ and value 1 with probability $p$ , then we use the MAP decoding strategy to minimize the block error rate and output the codeword closest to the channel output $\underline{y}$ . The expectation value of the code ensemble $P_B = \mathbb{E}P_B(\mathbb{C})$ is an indicator of code ensemble performances. We will show that, as $N\rightarrow \infty$ , codes with $l_{min}\geq 3$ will undergo a phase transition separating a low noise from a high noise phase. To derive a lower bound for the capacity of LDPC codes in a BSC channel, we take $\mathbb{N}=2^{NR}$ as the size of the codebook $\mathbb{C}$ and, by union bound:

$P_{B}(\mathbb{C})= \mathbb{P}\Big\{\exists \alpha \neq 0 \text{s.t. } d(\underline{x}^{(\alpha)},\underline{y})\leq d(\underline{0},\underline{y})\Big\}$ (22)

$\leq \sum_{\alpha=1}^{\textit{N}-1}\mathbb{P}\Big\{d(\underline{x}^{(\alpha)},\underline{y}) \leq d(\underline{0},\underline{y})\Big\}$ (23)

$\leq \sum_{w=1}^N \textit{N}(w)e^{-\gamma w}$ (24)

This derivation proves that the block error probability depends on the weight enumerator and the $exp(-\gamma w)$ . This second term shows that an increase in the weight of the codeword corresponds to their contribution being scaled down by an exponential factor. This is because it is less likely that the received message $\underline{y}$ will be closer to a codeword of large weight than to the all-zero codeword. A geometric construction of this phenomena implies that for large enough $p$ , Shannon’s Theorem implies that $P_B$ is bounded away from 0 for any non-vanishing rate $R > 0$ so that at any p less than the ML threshold for which the $lim_{N\rightarrow \infty}P_B=0$ , one can communicate with an arbitrarily small error probability. At a probability equal to the lower bound, the upper bound on $P_B$ is dominated by codewords of weight $w \approx N\Tilde{w}$ , suggesting that each time an error occurs, a finite fraction of the its are decoded incorrectly and that this fraction doesn’t change very much per transmission. The construction also illustrates that this fraction of incorrectly decoded bits jumps discontinuously from 0 to a finite value when $p$ crosses the critical value $p_{ML}$ , constituting a “gap.” This gap is close to a factor of 2.

Appendix C: BP performance

See figure 2 for an illustration of a factor graph illustrating this relationship. Again, recall that for LDPC code ensembles in large block-length limits, the degree distributions of variable nodes and check nodes are given by $\Lambda = {\Lambda_t}$ and $P = {P_k}$ respectively, where we assume that messages are initialized to $u_{a\rightarrow i}^{(0)} = 0$ for simplicity. This implies that the bit error probability is independent of the transmitted codeword and that therefore, we have the freedom to assume transmission of the all-zero codeword. In analyzing the recursion at the basis of the BP algorithm, we can show that decoding performance improves over time on the basis of symmetry and physical degradation.

Symmetry

Symmetry of channel log-likelihood and the variables appearing in density evolution are attributes of a desired BMS channel, suggesting that symmetry is preserved by BP operations in evolution. If we assume that the factor graph associated with an LDPC code is “tree-like”, we can apply BP decoding with a symmetric random initial condition and note that the messages $u_{a\rightarrow i}^{(t)}$ and $h_{i\rightarrow a}^{(t)}$ are symmetric variables at all $t\geq 0$ . This observance of symmetry is analogous to the Nishimori condition in spin glasses and holds for the MAP log-likelihood of a bit as well.

Physical degradation

Let’s first define physical degradation with BMS channels. If we take two channels BMS(1) and BMS(2) denoted by transition matrices

$\{Q_{1}(y|x)\}, \{Q_{2}(y|x)\}$ and output alphabets $\mathbb{Y}_1,\mathbb{Y}_2$ , then BMS(2) is physically degraded with respect to BMS(1) if there exists a third channel C with input alphabet $\mathbb{Y}_1,\mathbb{Y}_2$ such that BMS(2) is the concatenation of BMS(1) and C. If we represent transition matrix C as $\{R(y_2|y_1)\}$ we can represent the above physical degradation as

$Q_2(y_2|x) = \sum{y_1 \in \textit{Y}_1}R(y_2|y_1)Q_1(y_1|x)$ (25)

This is analogous to a Markov chain $X \rightarrow Y_1 \rightarrow Y_2$ following partial ordering. Channel reliability is then ordered by measures of conditional entropy and bit error rate. This extends to symmetric random variables, which are associated with BMS channels.

Thresholds

We then fix a particular LDPC code and look at BP messages as random variables due to randomness in the vector $\underline{y}$ with regards to the proposition down below, showing that the bit error rate decreases monotonously with time:

Proposition: If $B_{i,r}(F)$ is a tree, then $h_i^{(0)}\preceq h_i^{(1)} \preceq ... \preceq h_i^{(t-1)} \preceq h_i^{(t)}$ for any $t\leq r-1$ . Analogously, if $B_{i\rightarrow a,r}(F)$ , then $h_{i\rightarrow a}^{(0)}\preceq h_{i\rightarrow a}^{(1)} \preceq ... \preceq h_{i\rightarrow a}^{(t-1)} \preceq h_{i\rightarrow a}^{(t)}$ for any $t\leq r-1$ .

Density evolution in this manner is a useful estimate of the number of distributions of density evolution variables ${h^{(t)},u^{(t)}}$ and ${h_{*}^{(t)}}$ . By looking again at the bit error rate $P_{b}^{(t)}$ and the conditional entropy $H^{(t)}$ as both monotonically decreasing functions of the number of iterations and conversely, monotonically increasing functions of $p$ , we can derive a finite limit $P_b^{BP} \equiv \lim_{t\rightarrow\infty}P_b^{(t)}$ . The corresponding BP threshold can then be defined as

$p_{d} \equiv \sup \Big\{ p \in [0,1/2] : P_b^{BP}(p)=0 \Big\}$

For $p \leq p_d$ , however, increasing the number of iterations does not help as the bit error rate is asymptotically lower bounded by $P_{b}^{BP}(p)>0$ for a fixed number of iterations. Good LDPC codes are thus designed with a large BP threshold with design rate $R_{des}=1-P'(1)/\Lambda '(1)$ to maximize the threshold noise level for a given degree distribution pair. This ensemble will have a finite fraction of variable nodes of degree 2 and a large number of codewords with small weight, which ultimately prevent the block error probability from vanishing as $N\rightarrow \infty$ .

References

[1] Marc Mezard and Andrea Montanari.
Information, Physics, and Computation.
Oxford Graduate Texts, 2009.

	Interview about this… on Men in Computer Science
	Interview about this… on The Swiss Army Knife of C…
	Interview about this… on An alarming trend in K-12 Math…
	Interview about this… on The uneasy relationship betwee…
	Interview about this… on The different forms of quantum…

Algorithmic and Information Theoretic Decoding Thresholds for Low density Parity-Check Code

by Jeremy Dohmann, Vanessa Wong, Venkat Arun

Abstract

Introduction to Error Correcting Codes

Motivation

Low-density Parity Check Code

LDPC codebook geometry and $p_{c}$

Ease of construction

How the graphical model relates to fast decoding

Decoding Errors via Belief Propagation

Algorithm Overview

Algorithmic Thresholds for Belief Propagation (BP)

Definitions and notation

BP/Peeling algorithm

BP isn’t perfect

Phase transitions for BP

The solution space in $p_{d}\leq p \leq p_{c}$

The energy landscape of LDPC decoding

Energy minimization problem

MCMC and the relaxation time of a random walk

Exact values of the metastable energy states for the BEC

Appendix A: Random Code Ensembles

Appendix B: Weight enumerators and code performance

Appendix C: BP performance

Symmetry

Physical degradation

Thresholds

References

Published by Jeremy Dohmann

Leave a Reply Cancel reply

by Jeremy Dohmann, Vanessa Wong, Venkat Arun

Abstract

Introduction to Error Correcting Codes

Motivation

Low-density Parity Check Code

LDPC codebook geometry and

Ease of construction

How the graphical model relates to fast decoding

Decoding Errors via Belief Propagation

Algorithm Overview

Algorithmic Thresholds for Belief Propagation (BP)

Definitions and notation

BP/Peeling algorithm

BP isn’t perfect

Phase transitions for BP

The solution space in

The energy landscape of LDPC decoding

Energy minimization problem

MCMC and the relaxation time of a random walk

Exact values of the metastable energy states for the BEC

Appendix A: Random Code Ensembles

Appendix B: Weight enumerators and code performance

Appendix C: BP performance

Symmetry

Physical degradation

Thresholds

References

Share this:

Like this:

Related

Published by Jeremy Dohmann

Leave a Reply Cancel reply

LDPC codebook geometry and $p_{c}$

The solution space in $p_{d}\leq p \leq p_{c}$