Q7Digital Communication

(a) Shannon-Hartley Theorem and Its Implications

The Shannon-Hartley theorem establishes the theoretical maximum rate at which information can be transmitted over a communication channel with a specified bandwidth, in the presence of additive white Gaussian noise, with an arbitrarily small probability of error (given sufficiently sophisticated encoding/decoding, potentially requiring arbitrarily long code block lengths and correspondingly long processing delay to approach this limit in practice):

where C is the channel capacity (maximum achievable error-free information rate, in bits/second), B is the channel bandwidth (Hz), and S/N is the signal-to-noise power ratio (a linear, not dB, ratio) at the receiver.

Implications of the Shannon-Hartley Theorem

Fundamental capacity limit: the theorem establishes an absolute upper bound on the error-free information rate achievable over a given channel — no coding or modulation scheme, no matter how sophisticated, can reliably communicate at a rate exceeding C for the channel's given bandwidth and SNR; conversely, Shannon's theorem (the noisy-channel coding theorem, of which this capacity formula is a specific application to the Gaussian channel) also guarantees that rates arbitrarily close to (but not exceeding) C can in principle be achieved with an arbitrarily small error probability, given a sufficiently sophisticated (and potentially very long/complex) channel code.

Bandwidth-SNR trade-off: since capacity C depends on both B and S/N, a given desired capacity can in principle be achieved through different combinations of bandwidth and SNR — for example, doubling the bandwidth B while proportionally reducing S/N in a specific compensating way could, in some ranges, maintain the same overall capacity, illustrating that bandwidth and signal-to-noise ratio are, to a significant extent, interchangeable resources in achieving a target information rate (though the trade-off is not perfectly linear, since C depends logarithmically on S/N but linearly on B, making bandwidth generally a 'cheaper' resource to trade for capacity than SNR at high SNR values, and vice versa at very low SNR).

Finite capacity even with infinite bandwidth: a particularly important implication is that as bandwidth B increases toward infinity (with total signal power S held fixed, so that S/N₀B → 0 as B→∞, since the total noise power N=N₀B grows with bandwidth), the channel capacity does NOT increase without bound, but instead approaches a finite limiting value:

This shows that simply increasing bandwidth indefinitely, without also increasing signal power, yields diminishing and ultimately bounded returns in achievable capacity — a fundamental insight guiding the practical engineering trade-offs in wideband communication system design, spread-spectrum system design, and understanding the ultimate physical limits of any communication channel with a fixed total available signal power.

(b) Information Rate of the Telegraph Source

Given: dot duration = 0.2s; dash duration = 3×dot duration = 0.6s; P(dot) = 2×P(dash); time between symbols (gap) = 0.2s.

Step 1 — Find individual symbol probabilities: since P(dot)+P(dash)=1 and P(dot)=2P(dash), substituting: 2P(dash)+P(dash)=1, so 3P(dash)=1, giving P(dash)=1/3 and P(dot)=2/3.

Step 2 — Compute the source entropy (information per symbol):

Step 3 — Compute average symbol duration (including the inter-symbol gap): each transmitted symbol occupies its own duration plus the fixed inter-symbol gap of 0.2s:

Step 4 — Compute the information rate (entropy per symbol divided by average time per symbol):

Result: the information rate of the telegraph source is approximately 1.722 bits per second. This calculation illustrates the general principle that the information rate of a source with symbols of unequal duration and unequal probability must account for both the entropy (average information content per symbol, in bits) and the average time actually taken to transmit each symbol (including any fixed inter-symbol spacing/gap required by the physical signaling scheme), since a source transmitting highly probable (low-information) symbols quickly while less probable (high-information) symbols take proportionally longer to transmit will have an overall information rate that reflects this time-weighted average, rather than simply dividing the entropy by a single fixed symbol duration as would be appropriate only for a source with equal-duration symbols.

Cross-Check by Computing Information Contributed Per Symbol Type Separately

It is useful to verify the entropy calculation by considering the self-information carried by each symbol type individually rather than only via the combined entropy formula. The self-information of the dot is I(dot) = -log2(2/3) = log2(1.5) ≈ 0.585 bits, and the self-information of the dash is I(dash) = -log2(1/3) = log2(3) ≈ 1.585 bits — as expected, the less probable dash symbol carries more self-information than the more probable dot symbol, consistent with the fundamental principle that rarer events are inherently more 'informative' (more surprising) than common ones. Weighting each symbol's self-information by its probability of occurrence and summing:

This matches the H=0.9183 bits/symbol computed directly from the entropy formula in Step 2, confirming the calculation. It is also worth noting that this value is quite close to the maximum possible entropy of 1 bit/symbol for a binary-alphabet source (which would be achieved only if P(dot)=P(dash)=0.5) — the actual entropy of 0.9183 bits/symbol reflects a modest reduction below this theoretical maximum due to the source's symbol probabilities (2/3, 1/3) being unequal rather than uniform, illustrating the general property that any departure from an equiprobable symbol distribution necessarily reduces a source's entropy (average information content per symbol) below the maximum achievable value for that alphabet size, per the well-known result that entropy is maximized precisely when all symbols are equally likely.

Physical Interpretation and Relation to Source Coding

This telegraph source example is a classical illustration of Shannon's original motivating scenario for information theory, and it directly foreshadows the source-coding problem addressed later in this paper (Q.4 OR part (b), Shannon-Fano and Huffman coding). Just as a Huffman or Shannon-Fano code assigns shorter binary code words to more probable symbols and longer code words to less probable ones so as to minimize the average number of bits transmitted per symbol, this telegraph code (Morse-code-like in spirit) similarly assigns a shorter physical signal duration (0.2s) to the more probable dot and a longer duration (0.6s) to the less probable dash — in both cases, the underlying design principle is the same: efficient source representation matches shorter (cheaper, whether in bits or in time) representations to more frequently occurring symbols, and this matching is precisely what allows the information rate (bits of information conveyed per unit time) to approach the source's entropy bound as closely as the specific chosen code or signaling scheme allows.

It is also worth explicitly noting the role of the inter-symbol gap in this calculation: without accounting for the fixed 0.2s gap between symbols, the average transmission time per symbol would instead be computed as just P(dot)×Tdot + P(dash)×Tdash = (2/3)(0.2)+(1/3)(0.6) = 0.1333+0.2=0.3333s, which would yield a higher (and, in this problem, incorrect) apparent information rate of H/0.3333 ≈ 2.755 bits/second — nearly 60% higher than the correct 1.722 bits/second. This large discrepancy underscores how significant the fixed inter-symbol spacing overhead is in this particular problem (since the 0.2s gap is comparable in magnitude to the dot duration itself), and highlights, more generally, why practical information-rate calculations must always account for every component of the physical channel time actually consumed per transmitted symbol, not merely the nominal symbol/pulse duration itself.