Course Details
Subject {L-T-P / C} : EE4404 : Information Theory and Coding { 3-0-0 / 3}
Subject Nature : Theory
Coordinator : Prof. Dipti Patra
Syllabus
Entropy and Mutual Information: Entropy, Joint Entropy and Conditional Entropy, Relative Entropy and Mutual Information, Chain Rules, Data-Processing Inequality, Fano’s Inequality
Typical Sequences and Asymptotic Equipartition Property: Asymptotic Equipartition Property Theorem, Consequences of the AEP: Data Compression, High-Probability Sets and the Typical Set
Source Coding and Data Compression: Kraft Inequality, Huffman Codes, Optimality of Huffman Codes
Channel Capacity: Symmetric Channels, Properties of Channel Capacity, Jointly Typical Sequences, Channel Coding Theorem, Fano’s Inequality and the Converse to the Coding Theorem
Differential Entropy and Gaussian Channel: Differential Entropy, AEP for Continuous Random Variables, Properties of Differential Entropy, Relative Entropy, and Mutual Information, Coding Theorem for Gaussian Channels
Linear Binary Block Codes: Generator and Parity-Check Matrices, Repetition and Single-Parity-Check Codes, Binary Hamming Codes, Error Detection with Linear Block Codes, Weight Distribution and Minimum Hamming Distance of a Linear Block Code, Hard-decision and Soft-decision Decoding of Linear Block Codes, Cyclic Codes, Parameters of BCH and RS Codes, Interleaved and Concatenated Codes
Convolutional Codes: Encoder Realizations and Classifications, Minimal Encoders, Trellis representation, MLSD and the Viterbi Algorithm, Bit-wise MAP Decoding and the BCJR Algorithm.
Course Objectives
- Learn how to analyse and measure the information per symbol emitted from a source
- Learn how to analyse the information-carrying capacity of the communication channel
- Learn how to design source compression codes to improve the efficiency of information transmission.
- Learn the basic theory needed for data encryptions
Course Outcomes
At the end of the course, students will be able to <br />1. Understand and explain the basic concepts of information theory, source coding, channel and channel capacity, channel coding and relation among them. <br />2. Describe the real life applications based on the fundamental theory. <br />3. Calculate entropy, channel capacity, bit error rate, code rate, steady-state probability and so on. <br />4. Implement the encoder and decoder of one block code or convolutional code using any program language.
Essential Reading
- Thomas Cover, Joy Thomas, Elements of Information Theory, Wiley
- William Ryan, Shu Lin, Channel Codes: Classical and Modern, Cambridge
Supplementary Reading
- A. ElGamal and Y. H. Kim, Network Information Theory, Cambridge , 2011
- Robert Gallager, Information Theory and Reliable Communication, John Willey