Information Theory

This book provides an introduction to information theory, focussing on Shannon’s three foundational theorems of 1948–1949. Shannon’s first two theorems, based on the notion of entropy in probability theory, specify the extent to which a message can be compressed for fast transmission and how to erase errors associated with poor transmission. The third theorem, using Fourier theory, ensures that a signal can be reconstructed from a sufficiently fine sampling of it. These three theorems constitute