Probabilistic Graphical Models

Probabilistic Graphical Models (PGMs) are a class of statistical models that use a graph-based representation to encode the complex probabilistic relationships among a set of random variables. Within this framework, nodes represent the variables and edges signify conditional dependencies, allowing for a compact and intuitive visualization of a complex joint probability distribution. By merging graph theory with probability theory, PGMs provide a powerful system for reasoning and performing inference under uncertainty, with key examples including Bayesian Networks (using directed graphs) and Markov Random Fields (using undirected graphs).

1. Foundations of Probabilistic Graphical Models
   1. Core Concepts in Probability Theory
      1. Random Variables
         1. Discrete Random Variables
            1. Definition and Properties
            2. Sample Spaces for Discrete Variables
            3. Examples and Applications
         2. Continuous Random Variables
            1. Definition and Properties
            2. Sample Spaces for Continuous Variables
            3. Examples and Applications
         3. Probability Mass Functions (PMFs)
            1. Definition and Properties
            2. Normalization Conditions
            3. Computing Probabilities with PMFs
         4. Probability Density Functions (PDFs)
            1. Definition and Properties
            2. Normalization Conditions
            3. Computing Probabilities with PDFs
         5. Cumulative Distribution Functions (CDFs)
            1. Definition for Discrete Variables
            2. Definition for Continuous Variables
            3. Properties and Applications
      2. Probability Distributions
         1. Joint Probability Distribution
            1. Definition and Properties
            2. Representation for Multiple Variables
            3. Joint PMFs and PDFs
         2. Marginal Distribution
            1. Marginalization Process
            2. Marginalization for Discrete Variables
            3. Marginalization for Continuous Variables
            4. Law of Total Probability
         3. Conditional Probability Distribution
            1. Definition and Notation
            2. Conditional PMFs and PDFs
            3. Conditional Probability Tables (CPTs)
            4. Properties of Conditional Distributions
      3. Fundamental Probability Rules
         1. The Chain Rule of Probability
            1. Sequential Decomposition of Joint Distributions
            2. Application to Multiple Variables
            3. General Form for n Variables
         2. Bayes' Rule
            1. Derivation and Intuition
            2. Posterior, Prior, and Likelihood
            3. Applications in Inference
            4. Bayes' Rule for Multiple Variables
         3. Law of Total Expectation
            1. Definition and Applications
         4. Law of Total Variance
            1. Definition and Applications
      4. Independence and Conditional Independence
         1. Statistical Independence
            1. Definition for Two Variables
            2. Definition for Multiple Variables
            3. Properties of Independent Variables
         2. Conditional Independence
            1. Definition and Notation
            2. Properties of Conditional Independence
            3. Implications for Factorization
            4. Testing for Conditional Independence
   2. Core Concepts in Graph Theory
      1. Basic Graph Elements
         1. Nodes and Vertices
            1. Types of Nodes in Graphical Models
            2. Variable Nodes
            3. Factor Nodes
         2. Edges and Connections
            1. Directed Edges
            2. Undirected Edges
            3. Edge Weights and Labels
         3. Paths and Connectivity
            1. Simple Paths
            2. Directed Paths
            3. Undirected Paths
            4. Graph Connectivity
      2. Graph Types and Properties
         1. Directed vs. Undirected Graphs
            1. Definitions and Examples
            2. Implications for Model Semantics
            3. Mixed Graphs
         2. Cycles and Acyclicity
            1. Directed Cycles
            2. Undirected Cycles
            3. Directed Acyclic Graphs (DAGs)
            4. Properties of Acyclic Graphs
         3. Tree Structures
            1. Definition of Trees
            2. Spanning Trees
            3. Properties of Tree Graphs
      3. Graph Structural Concepts
         1. Cliques and Maximal Cliques
            1. Definition of Cliques
            2. Maximal Cliques
            3. Maximum Cliques
            4. Role in Factorization
         2. Graph Separation
            1. Separation in Undirected Graphs
            2. Cut Sets and Separating Sets
            3. D-separation in Directed Graphs
         3. Treewidth and Graph Decomposition
            1. Definition of Treewidth
            2. Tree Decomposition
            3. Computational Implications
   3. The Role of Graphical Models
      1. Motivation and Benefits
         1. Compact Representation of Joint Distributions
         2. Visualization of Dependencies
         3. Computational Advantages
      2. Encoding Distributions with Graphs
         1. Factorization via Graph Structure
         2. Independence Assumptions
         3. Semantic Interpretation
      3. Model Selection Considerations
         1. Expressiveness vs. Tractability Trade-off
         2. Model Complexity
         3. Computational Feasibility
         4. Overfitting and Underfitting