Deep Learning and Neural Networks

2.

Training Shallow Neural Networks

2.1.

Feedforward Neural Networks

2.1.1.

Definition and Structure

2.1.2.

Network Architecture Components

2.1.2.1.

2.1.2.1.1.

Role and Data Representation

2.1.2.1.2.

Feature Vector Processing

2.1.2.2.

2.1.2.2.1.

Purpose and Function

2.1.2.2.2.

Number of Layers and Neurons

2.1.2.2.3.

Representation Learning

2.1.2.3.

2.1.2.3.1.

Output Types for Regression

2.1.2.3.2.

Output Types for Classification

2.1.2.3.3.

Dimensionality Considerations

2.1.3.

Information Flow in Feedforward Networks

2.1.4.

Multilayer Perceptron (MLP)

2.1.4.1.

Structure and Properties

2.1.4.2.

Universal Approximation Theorem

2.1.4.3.

Depth vs. Width Trade-offs

2.2.

Measuring Performance: Loss Functions

2.2.1.

Purpose of Loss Functions

2.2.2.

Regression Loss Functions

2.2.2.1.

Mean Squared Error (MSE)

2.2.2.2.

Mean Absolute Error (MAE)

2.2.2.3.

2.2.3.

Classification Loss Functions

2.2.3.1.

Binary Cross-Entropy Loss

2.2.3.2.

Categorical Cross-Entropy Loss

2.2.3.3.

Sparse Categorical Cross-Entropy

2.2.3.4.

2.2.4.

Loss Function Selection Criteria

2.3.

The Learning Process: Gradient Descent

2.3.1.

The Concept of a Gradient

2.3.1.1.

Partial Derivatives

2.3.1.2.

Gradient Vector

2.3.1.3.

Direction of Steepest Ascent

2.3.2.

Gradient Descent Algorithm

2.3.2.1.

Iterative Optimization Process

2.3.2.2.

Parameter Update Rules

2.3.3.

2.3.3.1.

Impact on Convergence

2.3.3.2.

Choosing Appropriate Values

2.3.3.3.

Learning Rate Scheduling

2.3.4.

Batch Gradient Descent

2.3.4.1.

Full Dataset Processing

2.3.4.2.

Advantages and Disadvantages

2.3.5.

Stochastic Gradient Descent (SGD)

2.3.5.1.

Single Sample Updates

2.3.5.2.

Noise in Gradient Estimates

2.3.5.3.

Advantages and Disadvantages

2.3.6.

Mini-Batch Gradient Descent

2.3.6.1.

Batch Size Selection

2.3.6.2.

Trade-offs and Practical Considerations

2.4.

The Backpropagation Algorithm

2.4.1.

Overview of Backpropagation

2.4.2.

Mathematical Foundation

2.4.2.1.

The Chain Rule of Calculus

2.4.2.2.

Application in Neural Networks

2.4.3.

2.4.3.1.

Calculating Activations Layer by Layer

2.4.3.2.

Storing Intermediate Values

2.4.4.

2.4.4.1.

Computing Gradients

2.4.4.2.

Error Propagation

2.4.5.

Calculating Gradients for Weights and Biases

2.4.5.1.

Weight Gradient Computation

2.4.5.2.

Bias Gradient Computation

2.4.5.3.

Parameter Updates

2.4.6.

Computational Graphs

2.4.6.1.

Graph Representation of Networks

2.4.6.2.

Automatic Differentiation

Previous

1. Foundations of Machine Learning and Neural Networks

Go to top

Next

3. Deepening the Network