Transformer deep learning architecture

  1. Mathematical Foundations
    1. Linear Algebra Concepts
      1. Matrix Operations
        1. Matrix Multiplication
          1. Transpose Operations
            1. Batch Matrix Operations
            2. Vector Spaces
              1. Embedding Spaces
                1. Attention Score Spaces
                2. Dimensionality Considerations
                3. Probability and Information Theory
                  1. Probability Distributions
                    1. Softmax Distribution
                      1. Categorical Distribution
                      2. Information Measures
                        1. Entropy
                          1. Cross-Entropy
                            1. KL Divergence
                          2. Optimization Theory
                            1. Gradient-based Optimization
                              1. Convexity and Non-convexity
                                1. Local vs Global Optima
                                2. Computational Complexity
                                  1. Time Complexity Analysis
                                    1. Self-Attention Complexity
                                      1. Feed-Forward Complexity
                                      2. Space Complexity
                                        1. Memory Requirements
                                          1. Attention Matrix Storage

                                      Previous

                                      6. Training Methodology

                                      Go to top

                                      Next

                                      8. Architectural Analysis