Machine Learning

8.

Deep Learning and Neural Networks

8.1.

Neural Network Fundamentals

8.1.1.

Biological Inspiration

8.1.1.1.

Neurons and Synapses

8.1.1.2.

Action Potentials

8.1.1.3.

Learning in Biological Systems

8.1.2.

Artificial Neurons

8.1.2.1.

Mathematical Model

8.1.2.2.

Weighted Inputs

8.1.2.3.

8.1.2.4.

Activation Functions

8.1.3.

8.1.3.1.

Linear Classifier

8.1.3.2.

Perceptron Learning Rule

8.1.3.3.

8.1.3.4.

8.1.4.

Multi-Layer Perceptrons

8.1.4.1.

8.1.4.2.

Universal Approximation Theorem

8.1.4.3.

Network Architecture

8.1.4.4.

Depth vs. Width

8.2.

Activation Functions

8.2.1.

Linear Activation

8.2.1.1.

Identity Function

8.2.1.2.

8.2.2.

Sigmoid Functions

8.2.2.1.

Logistic Sigmoid

8.2.2.2.

Mathematical Properties

8.2.2.3.

Vanishing Gradient Problem

8.2.2.4.

8.2.3.

Hyperbolic Tangent

8.2.3.1.

Mathematical Definition

8.2.3.2.

Zero-Centered Output

8.2.3.3.

Comparison with Sigmoid

8.2.4.

Rectified Linear Unit

8.2.4.1.

ReLU Definition

8.2.4.2.

8.2.4.3.

Dead Neuron Problem

8.2.5.

8.2.5.1.

8.2.5.2.

Parametric ReLU

8.2.5.3.

Exponential Linear Unit

8.2.5.4.

8.2.5.5.

8.2.6.

Softmax Function

8.2.6.1.

Probability Distribution

8.2.6.2.

Multi-Class Classification

8.2.6.3.

Temperature Parameter

8.3.

Forward Propagation

8.3.1.

Layer-by-Layer Computation

8.3.2.

Matrix Operations

8.3.3.

8.3.4.

Computational Graphs

8.4.

Training Neural Networks

8.4.1.

8.4.1.1.

Regression Loss Functions

8.4.1.1.1.

Mean Squared Error

8.4.1.1.2.

Mean Absolute Error

8.4.1.1.3.

8.4.1.2.

Classification Loss Functions

8.4.1.2.1.

Binary Cross-Entropy

8.4.1.2.2.

Categorical Cross-Entropy

8.4.1.2.3.

Sparse Categorical Cross-Entropy

8.4.1.2.4.

8.4.1.2.5.

8.4.2.

Backpropagation Algorithm

8.4.2.1.

Chain Rule Application

8.4.2.2.

Gradient Computation

8.4.2.3.

Error Propagation

8.4.2.4.

Weight Update Rules

8.4.2.5.

Computational Efficiency

8.4.3.

Optimization Algorithms

8.4.3.1.

Gradient Descent Variants

8.4.3.1.1.

Batch Gradient Descent

8.4.3.1.2.

Stochastic Gradient Descent

8.4.3.1.3.

Mini-Batch Gradient Descent

8.4.3.2.

Momentum Methods

8.4.3.2.1.

Classical Momentum

8.4.3.2.2.

Nesterov Accelerated Gradient

8.4.3.3.

Adaptive Learning Rate Methods

8.4.3.3.1.

8.4.3.3.2.

8.4.3.3.3.

8.4.3.3.4.

8.4.3.3.5.

8.4.3.4.

Learning Rate Scheduling

8.4.3.4.1.

8.4.3.4.2.

Exponential Decay

8.4.3.4.3.

Cosine Annealing

8.4.3.4.4.

8.4.4.

Regularization Techniques

8.4.4.1.

8.4.4.1.1.

L2 Regularization

8.4.4.1.2.

Implementation Details

8.4.4.2.

8.4.4.2.1.

Random Neuron Deactivation

8.4.4.2.2.

Training vs. Inference

8.4.4.2.3.

Dropout Variants

8.4.4.3.

Batch Normalization

8.4.4.3.1.

Internal Covariate Shift

8.4.4.3.2.

Normalization Process

8.4.4.3.3.

Learnable Parameters

8.4.4.3.4.

8.4.4.4.

Layer Normalization

8.4.4.5.

Group Normalization

8.4.4.6.

8.4.4.6.1.

Validation Monitoring

8.4.4.6.2.

Patience and Restoration

8.5.

Convolutional Neural Networks

8.5.1.

Convolution Operation

8.5.1.1.

Mathematical Definition

8.5.1.2.

8.5.1.3.

Kernel/Filter Concepts

8.5.1.4.

Local Connectivity

8.5.2.

Convolutional Layers

8.5.2.1.

Multiple Filters

8.5.2.2.

Stride Parameter

8.5.2.3.

8.5.2.3.1.

8.5.2.3.2.

8.5.2.4.

8.5.3.

8.5.3.1.

8.5.3.2.

Average Pooling

8.5.3.3.

8.5.3.4.

Stride and Window Size

8.5.4.

CNN Architectures

8.5.4.1.

8.5.4.2.

8.5.4.3.

8.5.4.4.

8.5.4.5.

Inception Networks

8.5.4.6.

8.5.4.7.

8.5.5.

8.5.5.1.

Image Classification

8.5.5.2.

Object Detection

8.5.5.3.

Semantic Segmentation

8.5.5.4.

8.6.

Recurrent Neural Networks

8.6.1.

Sequential Data Processing

8.6.2.

Hidden State Concept

8.6.3.

8.6.3.1.

8.6.3.2.

8.6.3.3.

Backpropagation Through Time

8.6.4.

Vanishing and Exploding Gradients

8.6.4.1.

Problem Description

8.6.4.2.

Gradient Clipping

8.6.4.3.

Architectural Solutions

8.6.5.

Long Short-Term Memory

8.6.5.1.

8.6.5.2.

8.6.5.3.

8.6.5.4.

8.6.5.5.

8.6.6.

Gated Recurrent Units

8.6.6.1.

8.6.6.2.

8.6.6.3.

Simplified Architecture

8.6.7.

Bidirectional RNNs

8.6.7.1.

Forward and Backward Processing

8.6.7.2.

Context from Both Directions

8.6.8.

8.6.8.1.

Language Modeling

8.6.8.2.

Machine Translation

8.6.8.3.

Time Series Prediction

8.6.8.4.

Speech Recognition

8.7.

Advanced Architectures

8.7.1.

8.7.1.1.

Encoder-Decoder Structure

8.7.1.2.

Dimensionality Reduction

8.7.1.3.

Denoising Autoencoders

8.7.1.4.

Sparse Autoencoders

8.7.1.5.

Variational Autoencoders

8.7.2.

Attention Mechanisms

8.7.2.1.

Attention Concept

8.7.2.2.

Attention Weights

8.7.2.3.

Soft vs. Hard Attention

8.7.2.4.

8.7.3.

Transformer Architecture

8.7.3.1.

Multi-Head Attention

8.7.3.2.

Positional Encoding

8.7.3.3.

Encoder-Decoder Structure

8.7.3.4.

Layer Normalization

8.7.3.5.

Feed-Forward Networks

8.7.4.

Generative Adversarial Networks

8.7.4.1.

Generator Network

8.7.4.2.

Discriminator Network

8.7.4.3.

Adversarial Training

8.7.4.4.

Nash Equilibrium

8.7.4.5.

Training Challenges

8.7.4.6.

8.8.

Transfer Learning

8.8.1.

Pretrained Models

8.8.1.1.

ImageNet Pretrained Models

8.8.1.2.

Language Model Pretraining

8.8.1.3.

Domain Adaptation

8.8.2.

Fine-Tuning Strategies

8.8.2.1.

Feature Extraction

8.8.2.2.

Full Fine-Tuning

8.8.2.3.

Layer-Wise Learning Rates

8.8.2.4.

Gradual Unfreezing

8.8.3.

Domain Adaptation

8.8.3.1.

Source and Target Domains

8.8.3.2.

Distribution Shift

8.8.3.3.

Adaptation Techniques

Previous

7. Ensemble Methods and Advanced Techniques

Go to top

Next

9. Reinforcement Learning