Stochastic Processes

  1. Discrete-Time Markov Chains
    1. Fundamental Concepts
      1. State Space
        1. Finite State Spaces
          1. Countably Infinite State Spaces
          2. Transition Probabilities
            1. One-Step Transition Probabilities
              1. Transition Probability Matrix
                1. Row-Stochastic Property
                2. Markov Property
                  1. Memoryless Property
                    1. Formal Statement
                      1. Conditional Independence
                    2. Multi-Step Transitions
                      1. n-Step Transition Probabilities
                        1. Chapman-Kolmogorov Equations
                          1. Derivation
                            1. Matrix Formulation
                            2. Powers of Transition Matrix
                            3. Classification of States
                              1. Accessibility and Communication
                                1. Accessible States
                                  1. Communicating States
                                    1. Communication Relation
                                    2. Communicating Classes
                                      1. Equivalence Classes
                                        1. Closed Classes
                                          1. Open Classes
                                            1. Irreducible Chains
                                            2. Recurrence and Transience
                                              1. First Return Times
                                                1. Recurrent States
                                                  1. Transient States
                                                    1. Criteria for Recurrence
                                                    2. Positive and Null Recurrence
                                                      1. Expected Return Times
                                                        1. Positive Recurrent States
                                                          1. Null Recurrent States
                                                          2. Periodicity
                                                            1. Period of a State
                                                              1. Aperiodic States
                                                                1. Periodic Chains
                                                              2. Absorption and Hitting Probabilities
                                                                1. Absorbing States
                                                                  1. Absorption Probabilities
                                                                    1. First Passage Times
                                                                      1. Hitting Probabilities
                                                                        1. Fundamental Matrix
                                                                        2. Long-Run Behavior
                                                                          1. Limiting Distributions
                                                                            1. Existence Conditions
                                                                              1. Uniqueness Results
                                                                              2. Stationary Distributions
                                                                                1. Definition and Properties
                                                                                  1. Balance Equations
                                                                                    1. Calculation Methods
                                                                                    2. Ergodic Theorems
                                                                                      1. Ergodicity Conditions
                                                                                        1. Convergence Results
                                                                                          1. Time Averages
                                                                                        2. Special Classes and Applications
                                                                                          1. Time Reversibility
                                                                                            1. Detailed Balance Equations
                                                                                              1. Reversible Chains
                                                                                              2. Doubly Stochastic Matrices
                                                                                                1. Random Walks on Graphs
                                                                                                  1. Gambler's Ruin Problem
                                                                                                    1. Problem Formulation
                                                                                                      1. Solution Methods
                                                                                                        1. Probability of Ruin
                                                                                                        2. Branching Processes
                                                                                                          1. Galton-Watson Process
                                                                                                            1. Extinction Probabilities
                                                                                                              1. Critical Cases

                                                                                                          Previous

                                                                                                          2. Introduction to Stochastic Processes

                                                                                                          Go to top

                                                                                                          Next

                                                                                                          4. Poisson Processes