Deep Learning and Neural Networks

  1. Practical Considerations for Training
    1. Advanced Optimization Algorithms
      1. Limitations of Basic Gradient Descent
        1. Momentum-Based Methods
          1. Momentum
            1. Concept and Implementation
              1. Effect on Convergence
                1. Momentum Parameter Selection
                2. Nesterov Accelerated Gradient
                3. Adaptive Learning Rate Methods
                  1. AdaGrad
                    1. Adaptive Learning Rates
                      1. Accumulation of Squared Gradients
                        1. Strengths and Weaknesses
                        2. RMSprop
                          1. Running Average of Squared Gradients
                            1. Decay Rate Parameter
                              1. Use in Practice
                              2. Adam Optimizer
                                1. Combination of Momentum and RMSprop
                                  1. Bias Correction
                                    1. Hyperparameters
                                    2. AdamW
                                      1. Nadam
                                      2. Second-Order Methods
                                        1. Newton's Method
                                          1. Quasi-Newton Methods
                                            1. L-BFGS
                                          2. Regularization Techniques
                                            1. Weight Regularization
                                              1. L1 Regularization (Lasso)
                                                1. Mathematical Formulation
                                                  1. Sparsity Induction
                                                  2. L2 Regularization (Ridge)
                                                    1. Mathematical Formulation
                                                      1. Weight Decay Effect
                                                      2. Elastic Net Regularization
                                                      3. Dropout
                                                        1. Random Neuron Deactivation
                                                          1. Training vs. Inference Behavior
                                                            1. Dropout Rate Selection
                                                              1. Variants of Dropout
                                                              2. Early Stopping
                                                                1. Monitoring Validation Loss
                                                                  1. Patience Parameter
                                                                    1. Stopping Criteria
                                                                    2. Data Augmentation
                                                                      1. Techniques for Images
                                                                        1. Techniques for Text Data
                                                                          1. Techniques for Tabular Data
                                                                            1. Synthetic Data Generation
                                                                            2. Ensemble Methods
                                                                              1. Model Averaging
                                                                                1. Bagging
                                                                                  1. Boosting
                                                                                2. Hyperparameter Tuning
                                                                                  1. Importance of Hyperparameters
                                                                                    1. Hyperparameter Categories
                                                                                      1. Architecture Hyperparameters
                                                                                        1. Training Hyperparameters
                                                                                          1. Regularization Hyperparameters
                                                                                          2. Search Strategies
                                                                                            1. Grid Search
                                                                                              1. Exhaustive Search Strategy
                                                                                                1. Computational Complexity
                                                                                                2. Random Search
                                                                                                  1. Random Sampling of Hyperparameters
                                                                                                    1. Efficiency Advantages
                                                                                                    2. Bayesian Optimization
                                                                                                      1. Acquisition Functions
                                                                                                      2. Evolutionary Algorithms
                                                                                                      3. Learning Rate Scheduling
                                                                                                        1. Step Decay
                                                                                                          1. Exponential Decay
                                                                                                            1. Cosine Annealing
                                                                                                              1. Cyclical Learning Rates
                                                                                                                1. Warm Restarts
                                                                                                                2. Cross-Validation Strategies
                                                                                                                  1. K-Fold Cross-Validation
                                                                                                                    1. Stratified Cross-Validation
                                                                                                                      1. Time Series Cross-Validation

                                                                                                                  Previous

                                                                                                                  3. Deepening the Network

                                                                                                                  Go to top

                                                                                                                  Next

                                                                                                                  5. Convolutional Neural Networks (CNNs)