Deep Learning and Neural Networks

  1. Training Shallow Neural Networks
    1. Feedforward Neural Networks
      1. Definition and Structure
        1. Network Architecture Components
          1. Input Layer
            1. Role and Data Representation
              1. Feature Vector Processing
              2. Hidden Layers
                1. Purpose and Function
                  1. Number of Layers and Neurons
                    1. Representation Learning
                    2. Output Layer
                      1. Output Types for Regression
                        1. Output Types for Classification
                          1. Dimensionality Considerations
                        2. Information Flow in Feedforward Networks
                          1. Multilayer Perceptron (MLP)
                            1. Structure and Properties
                              1. Universal Approximation Theorem
                                1. Depth vs. Width Trade-offs
                              2. Measuring Performance: Loss Functions
                                1. Purpose of Loss Functions
                                  1. Regression Loss Functions
                                    1. Mean Squared Error (MSE)
                                      1. Mean Absolute Error (MAE)
                                        1. Huber Loss
                                        2. Classification Loss Functions
                                          1. Binary Cross-Entropy Loss
                                            1. Categorical Cross-Entropy Loss
                                              1. Sparse Categorical Cross-Entropy
                                                1. Hinge Loss
                                                2. Loss Function Selection Criteria
                                                3. The Learning Process: Gradient Descent
                                                  1. The Concept of a Gradient
                                                    1. Partial Derivatives
                                                      1. Gradient Vector
                                                        1. Direction of Steepest Ascent
                                                        2. Gradient Descent Algorithm
                                                          1. Iterative Optimization Process
                                                            1. Parameter Update Rules
                                                            2. Learning Rate
                                                              1. Impact on Convergence
                                                                1. Choosing Appropriate Values
                                                                  1. Learning Rate Scheduling
                                                                  2. Batch Gradient Descent
                                                                    1. Full Dataset Processing
                                                                      1. Advantages and Disadvantages
                                                                      2. Stochastic Gradient Descent (SGD)
                                                                        1. Single Sample Updates
                                                                          1. Noise in Gradient Estimates
                                                                            1. Advantages and Disadvantages
                                                                            2. Mini-Batch Gradient Descent
                                                                              1. Batch Size Selection
                                                                                1. Trade-offs and Practical Considerations
                                                                              2. The Backpropagation Algorithm
                                                                                1. Overview of Backpropagation
                                                                                  1. Mathematical Foundation
                                                                                    1. The Chain Rule of Calculus
                                                                                      1. Application in Neural Networks
                                                                                      2. Forward Pass
                                                                                        1. Calculating Activations Layer by Layer
                                                                                          1. Storing Intermediate Values
                                                                                          2. Backward Pass
                                                                                            1. Computing Gradients
                                                                                              1. Error Propagation
                                                                                              2. Calculating Gradients for Weights and Biases
                                                                                                1. Weight Gradient Computation
                                                                                                  1. Bias Gradient Computation
                                                                                                    1. Parameter Updates
                                                                                                    2. Computational Graphs
                                                                                                      1. Graph Representation of Networks
                                                                                                        1. Automatic Differentiation

                                                                                                    Previous

                                                                                                    1. Foundations of Machine Learning and Neural Networks

                                                                                                    Go to top

                                                                                                    Next

                                                                                                    3. Deepening the Network