Deep Learning and Neural Networks

  1. The Transformer Architecture
    1. Limitations of RNNs
      1. Sequential Processing Bottleneck
        1. Difficulty with Long-Range Dependencies
          1. Parallelization Challenges
            1. Computational Inefficiency
            2. The Attention Mechanism
              1. Motivation for Attention
                1. Attention as Soft Lookup
                  1. Query, Key, and Value Vectors
                    1. Mathematical Representation
                      1. Linear Transformations
                      2. Attention Score Computation
                        1. Scaled Dot-Product Attention
                          1. Computation Steps
                            1. Scaling Factor
                              1. Softmax Normalization
                              2. Attention Weights Interpretation
                              3. Self-Attention
                                1. Mechanism and Benefits
                                  1. Multi-Token Interactions
                                    1. Position-Aware Processing
                                      1. Computational Complexity
                                      2. Multi-Head Attention
                                        1. Parallel Attention Heads
                                          1. Different Representation Subspaces
                                            1. Concatenation and Projection
                                              1. Head Dimensionality
                                              2. The Transformer Architecture
                                                1. Overall Architecture Design
                                                  1. Encoder-Decoder Structure
                                                    1. Positional Encoding
                                                      1. Need for Position Information
                                                        1. Sinusoidal Positional Encoding
                                                          1. Learned Positional Embeddings
                                                          2. Encoder Block
                                                            1. Multi-Head Self-Attention
                                                              1. Layer Normalization
                                                                1. Feedforward Networks
                                                                  1. Residual Connections
                                                                  2. Decoder Block
                                                                    1. Masked Self-Attention
                                                                      1. Cross-Attention
                                                                        1. Autoregressive Generation
                                                                        2. Layer Normalization Placement
                                                                          1. Feedforward Networks
                                                                            1. Point-wise Operations
                                                                              1. Activation Functions
                                                                            2. Training Transformers
                                                                              1. Teacher Forcing
                                                                                1. Masked Language Modeling
                                                                                  1. Autoregressive Training
                                                                                  2. Transformer Variants
                                                                                    1. Encoder-Only Models
                                                                                      1. Decoder-Only Models
                                                                                        1. Encoder-Decoder Models
                                                                                        2. Applications and Impact
                                                                                          1. Machine Translation
                                                                                            1. Text Summarization
                                                                                              1. Question Answering
                                                                                                1. Large Language Models
                                                                                                  1. Pre-training Objectives
                                                                                                    1. Fine-Tuning Strategies
                                                                                                      1. Scaling Laws
                                                                                                        1. Emergent Abilities

                                                                                                    Previous

                                                                                                    6. Recurrent Neural Networks (RNNs)

                                                                                                    Go to top

                                                                                                    Next

                                                                                                    8. Generative Models