Information Theory

  1. Information Theory in Continuous Settings
    1. Differential Entropy
      1. Definition for Continuous Variables
        1. Relationship to Discrete Entropy
          1. Dependence on Coordinate System
            1. Negative Values
            2. Properties and Limitations
              1. Translation Invariance
                1. Scaling Properties
                  1. Maximum Differential Entropy
                  2. Differential Entropy of Common Distributions
                    1. Uniform Distribution
                      1. Gaussian Distribution
                        1. Exponential Distribution
                          1. Laplacian Distribution
                        2. Continuous Mutual Information
                          1. Definition and Properties
                            1. Invariance Under Invertible Transformations
                              1. Data Processing Inequality
                                1. Chain Rule
                                2. Estimation from Data
                                  1. Histogram Methods
                                    1. Kernel Density Estimation
                                      1. k-Nearest Neighbor Methods
                                    2. Gaussian Channels and Capacity
                                      1. AWGN Channel Model
                                        1. Signal and Noise Characteristics
                                          1. Power Constraints
                                            1. Bandwidth Constraints
                                            2. Capacity Calculations
                                              1. Single-Input Single-Output Channels
                                                1. Multiple-Input Multiple-Output Channels
                                                  1. Parallel Gaussian Channels
                                                  2. Optimal Input Distributions
                                                    1. Gaussian Inputs for AWGN
                                                      1. Water-Filling Solutions
                                                        1. Capacity-Achieving Strategies

                                                    Previous

                                                    6. Rate-Distortion Theory

                                                    Go to top

                                                    Next

                                                    8. Network Information Theory