Information Theory

  1. Source Coding and Lossless Compression
    1. Information Sources
      1. Discrete Memoryless Sources
        1. Definition and Properties
          1. Source Alphabet and Probabilities
            1. Entropy Rate
            2. Markov Sources
              1. First-Order Markov Sources
                1. Higher-Order Markov Sources
                  1. Stationary Sources
                  2. Ergodic Sources
                    1. Ergodicity and Stationarity
                      1. Asymptotic Mean Stationarity
                        1. Entropy Rate of Ergodic Sources
                      2. Fundamental Limits of Compression
                        1. Source Coding Theorem
                          1. Statement and Significance
                            1. Achievability Proof
                              1. Converse Proof
                              2. Asymptotic Equipartition Property
                                1. Weak AEP
                                  1. Strong AEP
                                    1. Typical Sequences
                                    2. Typical Sets
                                      1. Definition and Properties
                                        1. Probability of Typical Sets
                                          1. Size of Typical Sets
                                        2. Code Construction Theory
                                          1. Code Properties
                                            1. Uniquely Decodable Codes
                                              1. Prefix-Free Codes
                                                1. Suffix-Free Codes
                                                  1. Instantaneous Decodability
                                                  2. Kraft's Inequality
                                                    1. Statement for Prefix Codes
                                                      1. Proof and Implications
                                                        1. Converse of Kraft's Inequality
                                                        2. McMillan's Inequality
                                                          1. Extension to Uniquely Decodable Codes
                                                            1. Proof Technique
                                                              1. Relationship to Kraft's Inequality
                                                            2. Optimal Coding Algorithms
                                                              1. Huffman Coding
                                                                1. Algorithm Description
                                                                  1. Optimality Proof
                                                                    1. Average Code Length
                                                                      1. Limitations and Extensions
                                                                      2. Shannon-Fano Coding
                                                                        1. Top-Down Construction
                                                                          1. Performance Comparison
                                                                            1. Historical Significance
                                                                            2. Optimal Code Length Bounds
                                                                              1. Lower Bounds
                                                                                1. Upper Bounds
                                                                                  1. Redundancy Analysis
                                                                                2. Advanced Compression Techniques
                                                                                  1. Arithmetic Coding
                                                                                    1. Interval-Based Encoding
                                                                                      1. Precision and Implementation Issues
                                                                                        1. Adaptive Arithmetic Coding
                                                                                        2. Dictionary-Based Methods
                                                                                          1. LZ77 Algorithm
                                                                                            1. LZ78 Algorithm
                                                                                              1. LZW Algorithm
                                                                                                1. Performance Analysis
                                                                                                2. Universal Source Coding
                                                                                                  1. Lempel-Ziv Universal Coding
                                                                                                    1. Context Tree Weighting
                                                                                                      1. Prediction by Partial Matching
                                                                                                      2. Block Coding Extensions
                                                                                                        1. Extended Huffman Coding
                                                                                                          1. Tunstall Coding
                                                                                                            1. Asymptotic Optimality

                                                                                                        Previous

                                                                                                        2. Entropy and Information Measures

                                                                                                        Go to top

                                                                                                        Next

                                                                                                        4. Channel Capacity and Noisy Communication